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1.  Overview:  Neurotechnologies  for  In-Vehicle  Applications 


1.1  Objective 

This  report  provides  a  summary  of  the  work  performed  by  the  U.S.  Army  Research  Laboratory 
(ARL)  developing  neurotechnologies  for  in-vehicle  applications  for  the  work  period  of  fiscal 
year  2012,  the  first  year  of  the  research  program. 

1.2  Program  Overview 

Vehicle  survivability  is  an  important  issue  in  today’s  military.  However,  one  of  the  most  critical 
aspects  of  survivability  is  the  performance  of  Soldiers  operating  the  vehicles.  In  fact,  recent 
analyses  have  revealed  that  motor  vehicle  crashes  account  for  nearly  one-third  of  U.S.  military 
fatalities  annually,  including  both  privately  owned  and  military  motor  vehicles.  As  a  result, 
motor  vehicle  accidents  are  the  leading  cause  of  fatalities  among  U.S.  military  personnel  (Krahl 
et  al.,  2010).  Human  errors  are  to  blame  in  approximately  96%  of  all  civilian  crashes  (Amditis 
et  al.,  2010)  and  drowsiness  increases  the  driver’s  risk  of  a  crash  or  near-crash  by  at  least  a  factor 
of  four  (Klauer  et  al.,  2006).  These  effects  are  likely  to  be  accentuated  in  military  vehicle 
environments  because  Soldiers  perform  high-stress  missions  across  many  time  zones  around  the 
world  for  sustained  periods.  Often,  Soldiers  are  deprived  of  sleep  and  have  irregular  sleep/work 
patterns  (Mitler  et  al.,  1988).  Moreover,  unlike  most  civilian  and  commercial  driving,  which  is 
primarily  concerned  with  mobility  and  safety,  military  driving  involves  the  additional  concern  of 
security  (McDowell  et  al.,  2008). 

As  a  result,  ARL  has  undertaken  research  efforts  to  assess  the  state  of  human  vehicle  operators 
and  integrate  that  state  information  into  future  vehicle  systems.  ARL’s  work  will  focus 
specifically  on  developing  a  testbed  system  that  predicts  and  mitigates  task  performance 
decrements  to  optimize  Soldier-system  performance  in  realistic  operational  environments.  Initial 
research  will  be  focused  on  optimizing  performance  at  driving  tasks,  while  later  work  will 
attempt  to  generalize  algorithms  developed  for  driving  tasks  to  additional  Army-relevant  tasks. 

1.3  Approach 

ARL’s  approach  is  to  (1)  translate  current  brain-state  monitoring  and  performance  prediction 
technologies  to  real-world  domains,  by  evaluating  the  utility  of  the  current  technologies  in 
progressively  more  complex  and  realistic  conditions,  and  improve  the  technologies  as  needed  to 
develop  robust  brain  state  detection  and  task  performance  prediction,  and  (2)  enable  the  future 
fielding  of  these  performance  prediction  technologies  by  performing  the  research  to  develop 
increasingly  robust,  higher  resolution,  and  safe  portable  sensor  technologies,  while 
simultaneously  developing  improved  tools  for  evaluating  these  new  sensor  technologies. 


1 


1.4  Report  Overview 


The  remainder  of  this  report  will  highlight  ARL’s  accomplishments  in  the  first  year  of  the 
neurotechnologies  for  vehicle  applications  research  program.  The  report  is  organized  as  follows: 
section  2  will  briefly  review  state-of-the-art  techniques  for  fatigue-based  performance  prediction. 
Section  3  will  discuss  a  joint  ARL-U.S.  Army  Tank  Automotive  Research,  Development,  and 
Engineering  Center  (TARDEC)  experiment  for  translating  current  driver  performance  prediction 
technologies  to  real-world  driving  domains.  Section  4  will  describe  simulation  technologies 
developed  for  researching  driver  perfonnance.  Section  5  describes  ARL’s  work  on  developing 
task- independent  calibration  techniques  to  use  for  individual  Soldier  performance  prediction 
algorithms.  Section  6  will  describe  a  newly-funded  ARL  program  in  safe  and  wearable  sensor 
technologies.  Section  7  introduces  a  new  Small  Business  Innovative  Research  (SBIR)  program 
for  evaluating  electroencephalography  (EEG)  sensor  technologies.  Section  8  provides  an 
evaluation  of  existing  driver  performance  prediction  technologies. 

1.5  References 

Amditis,  A.;  Bimpas,  M.;  Thomaidis,  G.;  Tsogas,  M.;  Netto,  M.;  Mammar,  S.;  Beutner,  A.; 
Mohler,  N.;  Wirthgen,  T.;  Zipser,  S.;  Etemad,  A.;  Da  Lio,  M.;  Cicilloni,  R.  A  Situation- 
Adaptive  Lane-Keeping  Support  System:  Overview  of  the  SAFELANE  Approach.  IEEE 
Transactions  on  Intelligent  Transportation  Systems  2010,  11,  617-629. 

Klauer,  S.  G.;  Dingus,  T.  A.;  Neale,  V.  L.;  Sudweeks,  J.  D.;  Ramsey,  D.  J.  The  Impact  of  Driver 
Inattention  on  Near-Crash/Crash  Risk:  An  Analysis  Using  the  100 -Car  Naturalistic  Driving 
Study  Data\  DOT  HS-810  594;  Department  of  Transportation:  Washington,  DC,  2006. 

Krahl,  P.  L.;  Jankosky,  C.  J.;  Thomas,  R.  J.;  Hooper,  T.  I.  Systematic  Review  of  Military  Motor 
Vehicle  Crash-Related  Injuries.  Am.  J.  ofPrev.  Med.  2010,  38  (1),  S189-S196. 

McDowell,  K.;  Nunez,  P.;  Hutchins,  S.;  Metcalfe,  J.  S.  Secure  Mobility  and  the  Autonomous 
Driver.  IEEE  Transactions  on  Robotics,  2008,  24  (3),  688-697. 

Mitler,  M.  M.;  Carskadon,  M.  A.;  Czeisler,  C.  A.;  Dement,  W.  C.;  Dinges,  D.  F.;  Graeber,  R.  C. 
Catastrophes,  Sleep,  and  Public  Policy:  Consensus  Report.  Sleep  1988, 11  (1),  100. 
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2.  Reviewing  State-of-the-Art  Techniques  for  Fatigue-Based  Performance 
Prediction 


2.1  Introduction 

Degraded  Soldier  performance  is  an  essential  factor  that  can  undermine  U.S.  Army  efforts  aimed 
at  enhancing  vehicle  survivability.  In  particular,  Soldier  errors  can  lead  to  poor  tactical  decisions, 
which  in  turn  might  compromise  an  ongoing  mission.  In  the  case  of  military  operators  of  both 
unmanned  and  manned  vehicles,  degraded  performance  can  cause  otherwise  preventable 
accidents  that  might  render  a  vehicle  inoperable  or,  worse,  lead  to  injury  or  death.  Vehicle 
crashes  account  for  approximately  a  third  of  all  military  fatalities  each  year,  and  a  leading  cause 
of  motor  vehicle  crashes  is  fatigue.  Recent  fatigue  management  technologies  (FMTs)  geared 
toward  monitoring  and  mitigating  driver  performance  decrements  have  been  shown  to  improve 
driver  safety  and  survivability.  To  begin  addressing  this  issue,  ARL  will  require  an 
understanding  of  currently  available  FMTs,  as  well  as  an  understanding  of  what  technologies 
will  become  available  in  the  mid-term  and  long-term. 

Houser  et  al.  (2009),  for  example,  evaluated  the  costs  and  benefits  associated  with  lane  departure 
warning  systems  and  concluded  that  motor  carriers  purchasing  this  technology  would  likely 
prevent  thousands  to  tens  of  thousands  of  crashes  as  well  as  net  positive  returns  on  investments 
in  a  five-year  product  lifecycle.  For  military  applications,  it  is  likely  that  even  greater  benefits 
could  be  obtained,  in  part  because  standard  operating  procedures  could  be  implemented  to  ensure 
more  complete  integration  of  FMTs  into  all  aspects  of  Soldier  training,  mission  planning,  and 
operational  performance.  If  a  robust  and  reliable  FMT  were  to  be  deployed  with  crews  serving 
military  vehicles,  the  potential  impact  on  accident  reductions,  enhanced  situational  awareness, 
and  overall  crew  performance  could  be  quite  profound. 

2.2  Method 

ARL  has  conducted  a  literature  and  technology  review  to  identify  the  current  and  soon-to-be- 
available  FMTs  for  near-term  integration  into  active  safety  technology  programs  for  military 
vehicles.  Because  over  100  FMTs  are  either  currently  on  the  commercial  market  or  are  emerging 
from  various  stages  of  research  and  development,  we  have  focused  on  a  general  overview  of 
FMTs,  bolstered  with  specific  examples. 

2.3  Results  and  Discussion 

The  literature  review  shows  that  FMTs  can  be  generally  categorized  into  one  of  three  primary 
approaches:  (1)  vehicle-centered,  (2)  operator-centered,  or  (3)  combination/hybrid  approaches. 
Vehicle-centered  technologies  include  any  system  that  was  designed  to  utilize  data  from 
operator-vehicle  or  vehicle-environment  interactions,  such  as  steering  and  lane-keeping, 
accelerating,  braking,  and  encounters  with  objects  or  other  vehicles. 
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Operator-centered  technologies  include  any  system  that  utilizes  data  from  human  operators  and 
can  be  further  broken  into  three  subcategories:  (1)  biomathematical  models  of  alertness,  (2) 
fitness  for  duty  testing,  and  (3)  real-time  monitoring  technologies.  Finally,  hybrid  approaches 
incorporate  two  or  more  of  these  approaches  to  provide  a  more  comprehensive  and  robust  system 
than  a  single  technology  used  in  isolation. 

As  a  result,  we  conclude  that  a  phased  approach  is  likely  the  best  way  to  incorporate  these 
technologies.  Near-term  integration  efforts  should  focus  on  the  limited  number  of  currently 
available  and  proven  technologies,  such  as:  the  ASTiD*  (Fatigue  Management  International, 
Birkenhead  Wirral,  UK)  system,  which  integrates  a  biomathematical  model  of  fatigue  with  signal 
processing  of  steering  behavior;  the  OptalerC  (Optalert,  Melbourne,  Australia)  system,  which 
predicts  fatigue  based  on  eye-tracking  measures;  or  the  SafeTraK  (Takata  Corp.,  Tokyo,  Japan) 
system,  which  measures  lane  deviation  using  a  small  video  camera.  These  three  systems 
represent  an  important  combination  of  vehicle-  and  operator-centered  technologies  that  could 
account  for  a  broader  array  of  fatigue-induced  issues  than  any  one  current  system  could  manage 
on  its  own. 

In  a  second  medium-term  phase,  integration  of  these  types  of  systems  into  a  single  system,  such 
as  the  Cognitive  Avionics  Tool  Set  (CATS,  Operator  Performance  Laboratory,  University  of 
Iowa),  which  is  based  on  the  fusion  of  multisensory  operator-centric  data  streams,  would  enable 
increased  potential  for  achieving  greater  robustness  and  reliability  across  a  broader  set  of 
environments  and  operators.  Finally,  in  the  far-term,  continued  research  at  both  basic  and  applied 
levels  will  be  required  to  make  FMT  systems  achieve  higher  reliability,  validity,  sensitivity, 
specificity,  adaptability,  compatibility,  and  user  acceptability.  We  conclude  that  more  successful 
future  approaches  will  be  those  that  integrate  and  fuse  a  variety  of  data  at  each  level  from  human 
physiology  and  behavior  to  vehicle  motion  and  mission  performance.  A  more  in-depth  review  of 
this  work  can  be  found  in  (Kerick  et  al.,  in  press). 
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RRT-09-022;  Federal  Motor  Carrier  Safety  Administration:  Washington,  DC,  2009. 

Kerick,  S.;  Metcalfe,  J.;  Feng,  T.;  Ries,  A.;  McDowell,  K.  Review  of  Fatigue  Management 
Technologies  for  Enhanced  Military  Vehicle  Safety  and  Performance ,  U.S.  Army  Research 
Laboratory:  Aberdeen  Proving  Ground,  MD,  in  press. 


*  ASTiD  is  a  registered  trademark  of  Fatigue  Management  International,  Birkenhead  Wirral,  UK. 
’  Optalert  is  a  registered  trademark  of  Optalert,  Melbourne,  Australia. 
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3.  Translating  Driver  Performance  Prediction  Technologies  to  Real-World 
Driving  Domains 


3.1  Introduction 

The  Ground  Vehicle  Simulation  Laboratory  (GVSL)  at  the  U.S.  Army  Tank  Automotive 
Research,  Development  and  Engineering  Center  (TARDEC)  and  the  U.S.  Army  Research 
Laboratory  (ARL)/Human  Human  Research  and  Engineering  (HRED)  Translational 
Neuroscience  Branch  performed  a  collaborative  experiment,  funded  through  the  TARDEC  In- 
house  Laboratory  Independent  Research  program,  to  conduct  Soldier-centric  ground  vehicle 
research  on  the  utility  and  design  of  brain  computer  interaction  technologies  in  future  Army  crew 
stations.  In  particular,  the  goal  of  the  experiment  was  to  expand  upon  prior  work  published  by 
academic  partners  in  the  Army’s  Cognition  and  Neuroergonomics  Collaborative  Technology 
Alliance  by  investigating  the  capability  to  generalize  a  neurally-based  driver  performance 
prediction  methodology  (Lin  et  al.,  2005)  from  a  simple  single-task  laboratory  environment  to  a 
multitask  Army-relevant  driving  scenario.  To  collect  the  data,  the  project  utilized  TARDEC’s 
Ride  Motion  Simulator  (RMS)  and  its  real-time  simulation  environment  to  create  an  Army¬ 
relevant  task/scenario  and  environment  and  to  control  run-to-run  variability. 

This  experiment  will  provide  basic  research  and  proof-of-concept  results  to  support  the 
development  of  testbed  Warfighter  technologies  that  use  neural  signals  to  predict  and  mitigate 
task  performance  decrements  to  enhance  Soldier  performance  during  manned  ground  vehicle 
operations.  Moreover,  generalization  of  this  technology  has  significant  application  potential  to 
similar  long  Army-relevant  vigilance  tasks,  such  as  robotic  teleoperation  or  supervisory  control. 

The  method  extended  the  experiment  and  analysis  described  in  (Lin  et  al.,  2005)  to  a  more 
complex  and  realistic  Army-relevant  driving  task.  Lin  et  al.’s  (2005)  task  consisted  of  driving 
down  a  long  straight  traffic-free  road  on  a  six-degrees-of-freedom  ride  motion  platform. 
Periodically,  lateral  forces  were  applied  to  the  vehicle,  pushing  it  sideways  off  of  the  road.  By 
using  the  frequency  power- spectra  of  neural  signals  collected  through  electroencephalography 
(EEG),  Lin  et  al.  (2005)  were  able  to  predict  the  driver’s  reaction  time  to  the  lateral  perturbing 
force.  While  performance  of  this  system  at  this  task  is  quite  good,  it  is  not  clear  how  well  this 
technique  will  generalize  to  more  realistic  Army-relevant  environments.  We  have  extended  this 
research  through  enhancing  the  task  complexity  by  studying  driver  performance  as  part  of  a 
military  convoy.  Instead  of  only  one  vehicle  in  the  experiment,  there  was  also  an  automated 
convoy  leader  in  front  of  the  subject’s  vehicle  and  an  added  task  of  maintaining  proper  following 
distance  to  the  convoy  leader. 
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3.2  Method 

Fourteen  subjects  drove  down  a  very  long  straight  highway  during  a  single  45-min  driving 
session.  Subjects  were  tasked  with  staying  in  lane  and  maintaining  following  distance  to  the  lead 
vehicle.  Approximately  every  10  s  either  a  lateral  or  longitudinal  perturbation  occurred.  A  lateral 
perturbation  consisted  of  a  strong  lateral  force  pushing  the  vehicle  to  the  left  or  right.  A 
longitudinal  perturbation  consisted  of  either  a  sudden  slow  down  or  speed  up  by  the  lead  vehicle. 
Two  metrics  of  driver  performance  were  utilized.  The  first  was  defined  as  “lane  deviation,”  the 
distance  from  the  center  of  the  vehicle  to  the  center  of  the  driving  lane.  The  second  was  defined 
as  “following  distance  deviation,”  the  distance  from  the  center  of  the  vehicle  to  the  following 
distance  the  subjects  were  required  to  maintain.  Both  performance  metrics  were  measured 
continuously  at  approximately  100  Hz.  Throughout  the  driving  session,  64-channel  EEG  signals 
were  recorded  at  2048  Hz. 

3.3  Discussion 

Data  collection  has  been  completed  and  data  analysis  is  currently  in  progress.  TARDEC  is 
analyzing  the  behavioral  data  (e.g.,  steering  behavior  and  following  distance)  collected  during 
the  experiment  looking  for  correlations  between  driver  reactions  to  the  longitudinal  and  lateral 
perturbations.  Simultaneously,  ARL  is  analyzing  the  neural  data,  attempting  to  predict  Soldier 
reaction  time  to  the  lateral  and  longitudinal  perturbations  using  Lin  et  al.’s  (2005)  algorithm  on 
the  EEG  data.  In  addition  to  this  primary  experiment,  ARL  also  provided  expertise  and  support 
for  the  analysis  of  a  joint  GVSL  and  Joint  Program  Office-Mine  Resistant  Ambush  Protected 
Vehicles  experiment  focused  on  identifying  driver  task  load  in  real  time. 

3.4  References 

Lin,  C.  T.;  Wu,  R.  C.;  Jung,  T.  P.;  Liang,  S.  F.;  Huang,  T.  Y.  Estimating  Driving  Performance 
Based  on  EEG  Spectrum  Analysis.  EURASIP  J.  on  Appl.  Sig.  Proc.  2005, 19,  3165-3174. 

Lin,  C.  T.;  Wu,  R.  C.;  Liang,  S.  F.;  Huang,  T.  Y.;  Chao,  W.  H.;  Chen,  Y.  J.;  Jung,  T.  P.  EEG- 
Based  Drowsiness  Estimation  for  Safety  Driving  Using  Independent  Component  Analysis. 
IEEE  Trans.  Circ.  Syst.  2005b,  52,  2726-2738. 
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4.  Simulation  Platform  for  Developing  and  Evaluating  Neurotechnologies  for 
In-Vehicle  Applications 


4.1  Introduction 

The  U.S.  Army  Research  Laboratory  (ARL)  contracted  the  DCS  Corporation  to  develop  a 
flexible  simulation  testbed  system  for  testing  driver  performance  prediction  for  basic  science 
experiments  and  neurotechnology  system  development.  The  initial  system  consists  of  two 
primary  components:  (1)  a  simulation  environment  that  is  user-configurable,  allowing  for  a  large 
variety  of  scenarios  to  explore  vehicle  driving  performance,  and  (2)  an  Arduino-based  system 
that  synchronizes  simulation  events  and  subject  responses  with  physiological  indicators  such  as 
electroencephalography  (EEG)  and  eye-tracking. 

4.2  Configurable  Simulation  System 

The  simulation  system  is  based  on  the  SimCreator  system  (Real  Time  Technologies,  Dearborn, 
MI)  to  provide  real-time  simulation  of  driving  tasks.  The  configurability  of  the  system  arises 
from  a  tile-based  paradigm  that  generates  a  driving  terrain  database  of  any  desired  length  by 
assembling  a  number  of  road  tiles  (see  figure  1). 


Figure  1 .  Example  road  tiles  for  the  driving  simulation  system. 

Each  tile  has  one  or  more  four-lane  road  segments  (the  left  two  are  oncoming  and  the  right  two 
are  in  the  direction  of  travel)  where  the  road  segments  enter  and  exit  each  tile  in  the  center  of  the 
edge.  Each  tile  has  been  carefully  surveyed  to  allow  the  real-time  provision  of  standard  driving 
performance  metrics,  such  as  lane  deviation,  time-to-contact,  etc.  Detailed  data  logging  provides 
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a  capability  to  more  thoroughly  assess  these  and  other  driving  performance  metrics  offline 
during  post-tests  analyses.  In  addition  to  the  road  tiles,  the  system  allows  the  user  to  configure 
several  driving  events  within  the  tiles.  Such  events  currently  include  wind  perturbations  that  push 
the  vehicle  sideways  off  the  road;  speed  limit  signs;  vehicles  driving  in  the  oncoming  lanes,  in 
the  direction  of  travel,  or  across  the  road  at  intersections;  pedestrian  traffic;  and  construction 
zones.  For  each  event,  the  user  can  control  the  number  of  entities,  event  onset  and  offset,  event 
location,  and  the  motion  path  (for  moving  entities). 

To  support  the  broader  goals  of  allowing  basic  science  experiments  and  neuro technology  design, 
the  simulation  system  was  designed  from  the  ground  up  for  integration  with  standard 
physiological  data  acquisition  systems,  such  as  EEG,  eye-tracking,  electrocardiography  (EKG), 
electrodermal  activity  (EDA),  and  respiration.  All  signals  are  synchronized  using  an  Arduino- 
based  synchronization  system  and  recorded  on  locally  networked  PCs. 

4.3  Arduino-Based  Simulation  Synchronization 

Time  synchronization  across  measurement  devices  in  neuroscience  experiments  is  critical  for  the 
analysis  and  interpretation  of  datasets,  in  particular  when  performing  EEG  studies.  Currently 
available  commercial  options  are  expensive  and  often  limited  to  the  integration  of  specific 
sensors.  The  aim  of  this  project  was  to  develop  a  general-purpose  low-cost  synchronization 
solution  that  could  support  integration  of  a  wide  variety  of  digital  and  analogue  sensors  with 
submillisecond  time  resolution. 

The  resulting  temporal  synchronization  device  (TSD)  is  built  on  the  Arduino  Mega  platform,  a 
low-cost  open-source  easy-to-use  microcontroller,  in  conjunction  with  a  custom-built  Arduino 
Shield  expansion  card.  This  device  provides  a  general-purpose  solution  for  synchronizing 
multiple  supplementary  sensors  with  physiological  data.  The  device  works  by  sampling  all  of  the 
sensors  it  is  integrating  as  well  as  a  pulse  signal  sent  by  the  EEG  system.  It  integrates  the 
collected  data  with  the  pulse  signal  and  streams  it  to  a  host  computer  via  a  USB  serial 
connection.  The  current  sensor  setup  uses  23  of  the  TSD’s  input  ports  and  streams  the  data  to  the 
host  machine  at  1024  Hz.  As  a  result,  the  TSD  could  integrate  additional  sensors  without 
dropping  the  level  of  synchronization  accuracy. 

The  TSD  provides  a  general-purpose  low-cost  solution  for  synchronizing  data  and  time  triggers 
from  multiple  sources  with  a  high  degree  of  accuracy,  allowing  for  the  integration  of  multiple 
data  collection  devices  with  differing  types  of  stimuli,  including  stimuli  presented  in  generally 
harder- to-synchronize  domains  such  as  virtual  environments  or  simulation  engines. 

4.4  Conclusion 

In  total,  the  system  provides  the  capability  for  formal  data  collection  events,  either  for  basic 
science  experiments  or  as  technology  assessment  and  prototyping.  All  data,  including  driving 
performance,  simulation  behaviors  (i.e.,  entity-  and  tile-based  information),  and  physiological 
measures,  are  synchronized  to  enable  precise  assessment  of  relationships  between  brain  and 
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behavior.  In  addition,  the  system  is  completely  configurable  such  that  analyst-level  users  may 
completely  design  test  scenarios  through  an  automated  software  toolset.  Options  include 
specification  of  tile  sequences,  event  dynamics,  as  well  as  custom  user-specified  events.  Overall, 
such  a  system  allows  for  the  specification  of  experiment  requirements  of  a  broad  range  in  terms 
of  levels  of  complexity  (from  simple  straight  roads  with  no  traffic  to  a  complex  urban 
environment  with  both  vehicle  and  pedestrian  traffic),  while  maintaining  an  ability  for 
experimenters  to  derive  highly  precise  data  regarding  operator  performance  and  psycho- 
physiological  state. 
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5.  Task-Independent  Calibration  Techniques  for  Individual  Soldier 
Performance  Prediction  Algorithms 


5.1  Introduction 

There  are  numerous  challenges  to  effectively  using  neural  signals  in  brain-computer  interaction 
(BCI)  technology  development.  One  of  the  primary  challenges  lies  with  the  individual 
differences  in  neural  or  physiological  response  to  tasks  or  stimuli.  In  order  for  the  performance 
prediction  algorithms  described  in  section  7  of  this  report  to  address  these  individual  differences, 
a  calibration  period  of  training  data  needs  to  be  collected  from  each  individual.  However,  the 
time  spent  collecting  this  data  is  likely  to  decrease  the  utility  of  these  systems,  slowing  their  rate 
of  acceptance. 

Currently,  Lin  et  al.’s  (2005a)  algorithm  requires  a  calibration  period  of  at  least  15  min  of 
driving  in  a  simulated  environment  to  obtain  the  training  data  needed  for  predicting  driver 
performance.  However,  this  is  a  less  than  ideal  solution.  If  Soldiers  could  use  a  general  handheld 
calibration  app  for  a  few  minutes  instead,  training  data  acquisition  would  require  considerably 
less  expense.  In  addition,  it  is  more  likely  that  this  calibration  solution  will  generalize  to 
additional  tasks  beyond  driving 

5.2  Methods 

The  U.S.  Army  Research  Laboratory  (ARL)  has  implemented  two  general  calibration  tasks  into 
an  Android  device.  These  two  tasks  are  the  continuous  tracking  task  (CTT)  and  the  psychomotor 
vigilance  task  (PVT),  both  of  which  have  prior  research  showing  that  performance  on  these  tasks 
may  be  useful  for  predicting  performance  at  driving  tasks.  It  has  been  shown  that  the  same  data 
processing  methods  for  predicting  performance  at  the  Lin  et  al.  (2005b)  driving  task  are  also 
effective  at  predicting  performance  on  the  CTT  task  (Makeig  and  Jolley,  1995).  The  CTT 
consists  of  performing  a  “bubble  level”  style  task,  where  the  user  attempts  to  tilt  the  Android 
device  to  keep  a  small  moving  circle  inside  of  a  larger  static  circle.  Periodically,  a  random 
perturbing  force  is  applied  to  the  small  moving  bubble  circle,  pushing  it  out  of  place.  The 
Android  CTT  app  measures  and  records  user  score  and  response  times.  PVT  performance  has 
been  shown  to  be  closely  associated  with  the  fatigue  level  of  the  user,  which  can  be  used  to 
predict  performance  at  driving  tasks  (Dinges  and  Powell,  1985;  Balkin,  2000).  The  PVT  consists 
of  a  basic  vigilance  task,  where  the  user  watches  the  screen  of  the  device  looking  for  a  rarely 
occurring  stimulus.  When  the  users  see  the  stimulus,  they  respond  by  touching  the  screen.  The 
PVT  app  also  measures  and  records  user  score  and  response  times. 

5.3  Discussion 

The  implementation  of  the  CTT  and  the  PVT,  and  recording  score  and  response  times  on  an 
Android  device,  has  been  successful.  Current  work  focuses  on  implementing  EEG  recording 
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capabilities  for  the  Android  platform  and  synchronizing  those  capabilities  with  the  CTT  and  PVT 
apps.  In  the  future,  we  will  use  the  recorded  EEG  and  behavioral  data  during  the  CTT  and  PVT 
tasks  to  evaluate  their  utility  as  calibration  mechanisms  for  predicting  performance  at  driving 
tasks. 
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6.  Safe  and  Wearable  Sensor  Technology 


6.1  Summary 

6.1.1  Overview 

There  is  extremely  strong  interest  in  moving  neurotechnology  out  of  laboratory  settings,  and  into 
real-world  every-day  environments  where  behavior  can  be  measured  in  its  native  setting.  Support 
for  the  importance  of  this  idea,  and  the  extensive  benefits  gained  from  it,  have  come  from  as  high 
as  the  Office  of  the  President  (National  Science  and  Technology  Council,  2009).  Meanwhile,  the 
National  Research  Council  has  strongly  encouraged  the  U.S.  Army  to  develop  sensor  arrays 
capable  of  accurately  detecting  and  quantifying  a  Soldier’s  physiological  and  cognitive  states 
while  performing  mission-related  tasks  in  a  field  environment  (National  Research  Council, 

2009).  One  promising  method  for  doing  this  is  through  electroencephalography  (EEG),  a 
noninvasive  method  to  measure  brain  activity,  as  characterized  by  the  electrical  potentials 
observed  at  the  scalp  that  provide  high  temporal  resolution  consistent  with  the  time  scales  of 
brain  processes.  EEG  is  the  most  likely  method  for  developing  technologies  based  on  brain 
activity  due  to  the  low  cost  and  portability  compared  to  alternative  brain-imaging  technologies 
such  as  magnetoencephalography  (MEG)  and  functional  magnetic  resonance  imaging  (fMRI) 
systems,  both  of  which  require  a  room-sized  device  that  costs  hundreds  of  thousands  of  dollars 
and  a  completely  stationary  user. 

However,  current  EEG  technologies  are  time  consuming  to  prepare,  cannot  be  placed  in  vehicles 
due  to  the  lack  of  portability  owing  to  the  necessity  for  the  electrodes  to  be  physically  tethered  to 
a  signal  amplifier,  and  the  metal  electrodes  may  present  safety  concerns  in  test,  research,  and 
evaluation  environments  where  dynamic  body  motions  and  impact  events  may  occur.  For 
example,  typical  laboratory  EEG  systems  involve  attaching  as  many  as  256  small  tethered  metal 
(e.g.,  silver  chloride)  electrodes  to  the  scalp  as  well  as  the  application  of  conductive  gels  to 
reduce  skin-electrode  contact  impedance  in  order  to  obtain  adequate  signal  quality.  As  a  result, 
integration  of  BCI  technologies  with  in-vehicle  systems  is  severely  challenged  by  the  form  factor 
of  contemporary  sensory  systems.  In  order  to  avoid  these  challenges,  vehicle  system  developers 
will  require  easily  wearable  EEG  systems  that  are  wireless,  comfortable,  robust,  safe,  and 
validated  to  work  within  a  number  of  environments. 

Currently,  the  U.S.  Army  Research  Laboratory  (ARL)  is  addressing  this  need  through  research 
focusing  on  two  specific  aspects  of  wearable  EEG  technology:  (1)  dry,  comfortable,  impact-safe 
electrodes  with  flexible  biosensing  electronics,  and  (2)  evaluation  technologies  to  serve  as  a 
standard  for  testing  new  hardware  approaches. 
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6.1.2  Dry,  Impact-Safe  EEG  Electrodes 


In  2011,  ARL  and  Cognition  and  Neuroergonomics  (CaN)  Collaborative  Technology  Alliance 
(CTA)  partner  National  Chiao-Tung  University  (NCTU)  in  Taiwan  developed  a  novel  EEG 
sensor  design  comprised  of  a  polymer  foam  substrate  covered  by  a  conductive  fabric  (Lin  et  al., 
2011,  figure  2).  The  foam-based  sensors  were  shown  to  provide  signal  quality  comparable  to 
traditional,  laboratory- standard  wet  (gelled)  electrodes  while  having  the  benefits  of  easy  and  fast 
application,  high  conformity  with  the  scalp  surface,  improved  comfort  and  performance  during 
long-term  wear  (about  5  h),  and  the  potential  to  provide  increased  safety  in  dynamic 
measurement  conditions.  The  authors  also  indicate  that  the  sensor  technology  is  cost-effective, 
with  per  unit  costs  for  sensor  fabrication  of  about  U.S.  $0.30. 


Figure  2.  Dry,  conductive-fabric  EEG  electrode. 

The  performance  of  the  foam-based  EEG  sensors  was  then  evaluated  in  a  standard  laboratory 
task  and  in  a  brain-computer  interaction  technology  (Liao  et  al.,  2012a)  that  allowed  real-time 
control  of  a  simple  video  archery  game.  The  accuracy  of  shots  in  the  game  was  controlled  by  the 
users’  ability  to  produce  a  “mentally  focused”  state,  which  was  operationally  defined  based  on 
previous  research  results  as  the  inverse  of  the  average  EEG  spectral  power  in  the  8-  to  12-Hz 
frequency  range. 

In  addition  to  the  development  of  these  sensor  technologies,  NCTU,  ARL,  and  University  of 
California  San  Diego  (UCSD)  partners  also  provided  a  survey  of  current  biosensor  technologies 
and  provided  expert  analysis  and  prediction  of  future  BCI  research  and  development  efforts 
(Liao  et  al.,  2012b). 
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6.1.3  Flexible  Biosensing 

In  order  to  be  effective  and  comfortable,  EEG  systems  must  stretch  with  Soldier  motions  without 
hindering  movement  or  otherwise  negatively  impacting  the  Soldier’s  mission  or  safety.  Standard 
rigid  electronics  have  well-understood  signal  filtering  characteristics  but  significantly  decrease 
the  flexibility  and  comfort  of  electronic  sensors  when  in  close  proximity  to  the  body.  When 
deployed  in  wearable  systems,  rigid  components  can  also  compromise  survivability,  particularly 
if  employed  inside  protective  armor.  Pliability  of  the  material  solution  is  important  for  increased 
Soldier  comfort  and  safety  on  the  battlefield. 

To  address  this  issue,  ARL,  through  collaboration  among  its  Human  Research  and  Engineering 
Directorate  (HRED),  Weapons  and  Materials  Research  Directorate  (WMRD),  and  Vehicle 
Technologies  Directorate  (VTD),  has  initiated  a  new  program  focused  on  the  development  of 
pliable,  stretchable  conductive  materials  that  have  highly  predictable  and  stable  conductance 
over  a  range  of  deformation.  During  fiscal  year  2012  (FY12),  this  group  manufactured  a 
prototype  array  of  stretchable  EEG  electrode  wire  conductors  in  a  format  similar  to  that  of  the 
Advanced  Brain  Monitoring  X-10  EEG  system  (figure  2).  These  stretchable  wires  were 
embedded  in  a  stretchable  acrylic  coating  with  exposed  ends  for  connecting  sensor  pads.  This 
prototype  yielded  positive  results,  demonstrating  comparable  signal  attenuation  to  a  state-of-the- 
art  laboratory  EEG  lead  when  under  20%  strain. 

However,  one  pitfall  discovered  in  this  initial  approach  was  that  the  nonconductive  outer  coating 
of  the  stretchable  wire  array  poses  a  barrier  to  direct  contact  with  the  wearer  (which  is  typically 
bridged  via  electrolytic  gel).  Follow-on  efforts  showed  that  one  viable  approach  is  to  add  a 
crimp-on  style  metal  snap  that  penetrates  the  material  and  provides  media  for  connecting  a 
separate  electrode  connector  matching  the  format  of  the  dry,  impact-safe  EEG  electrodes  under 
development  through  the  CaN  CTA.  Initial  results  indicate  that  this  method  provides  an 
acceptable  conductive  connection  but  additional  testing  is  needed  to  determine  whether  or  not 
the  crimp  will  lead  to  tearing  of  the  stretchable  material  with  repeated  use.  Also  during  this  fiscal 
year,  funding  was  applied  for  and  granted  through  the  ARL  Director’s  Research  Initiative  (DRI) 
program,  securing  support  for  FY13  through  FY 15.  A  more  in-depth  review  of  this  work 
follows. 

6.2  Introduction 

There  is  a  current  limitation  in  our  ability  to  assess  cognitive  and  other  brain  functions  within 
real-world,  dynamic  Army  environments.  This  is  due  in  part  to  a  lack  of  tools  that  are  logistically 
pragmatic  or  even  safe  to  use  in  operational  settings.  Recent  advances  in  materials  science 
leading  to  conductive  yet  pliable  and  stretchable  materials  suggest  the  opportunity  for 
revolutionizing  the  types  of  hardware  that  may  be  used  for  neurocognitive  assessment  in  the 
future.  This  report  describes  the  background,  initial  efforts,  and  near-term  plans  of  a  new  effort 
(begun  in  FY12)  for  leveraging  these  advances  for  developing  stretchable,  deformable 
conductors  and  electrode  interfaces  to  be  used  in  EEG. 
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6.3  Background 

The  National  Research  Council  has  strongly  encouraged  multiple  branches  of  the  U.S.  armed 
forces  to  develop  sensor  arrays  capable  of  accurately  detecting  and  quantifying  a  Soldier’s 
physiological  and  cognitive  states  while  performing  mission-related  tasks  in  a  field  environment 
(National  Research  Council,  2009;  National  Research  Council,  2012).  In  order  to  be  effective 
and  comfortable,  such  arrays  must  accommodate  the  full  range  of  Soldier  motions  without 
hindering  movement  or  otherwise  negatively  impacting  the  mission  or  safety  of  the  Soldier. 
Additionally,  they  must  be  pliable  and  deformable  to  mitigate  potential  safety  concerns  from 
ballistics  or  other  impact  injuries.  Meanwhile,  most  biosensing  modalities  such  as 
electrocardiography  (EKG)  or  EEG  involve  measuring  extremely  small  voltages  at  the  skin¬ 
sensor  interface  (e.g.,  pV  for  EEG  or  mV  for  EKG),  necessitating  a  sensitivity  to  miniscule 
changes  in  amplitude.  As  a  result,  going  from  the  “bench  to  the  battlefield”  requires  novel 
technologies  to  observe  bioelectronic  activity  in  real-world  environments.  Over  the  past  five 
years,  foundational  steps  have  been  taken  toward  this  end  through  funding  by  ARL,  the  National 
Institutes  of  Health  (NIH),  the  Defense  Advanced  Research  Projects  Agency  (DARPA),  and 
private  industry  (Liao  et  al.,  2012b).  Even  with  these  investments,  significant  conceptual 
progress  in  both  hardware  and  software  needs  to  be  accomplished  to  develop  suitable  capabilities 
to  enable  measurement  of  brain  and  body  function  in  Warfighter  operational  environments. 

The  combat  environment  is  a  unique  application  for  biosensing,  which  poses  challenges  to  safety 
and  comfort  not  encountered  in  other  circumstances.  According  to  personnel  in  ARL’s  WMRD 
who  design  prototype  helmet  systems,  a  combat-ready  EEG  system  must  be  made  of 
reconfigurable  deformable  components  inside  of  the  helmet  system.  To  the  best  of  the  authors’ 
knowledge,  these  two  features  are  not  part  of  any  current  development  projects  by  any  agency. 

Standard  rigid  electronics,  such  as  stranded-fiber  or  stamped-metal  conductors,  have  well- 
understood  signal  filtering  characteristics  but  significantly  decrease  the  flexibility  and  comfort  of 
electronic  sensors  when  in  close  proximity  to  the  body.  When  deployed  in  wearable  systems, 
rigid  components  can  also  compromise  survivability,  particularly  if  employed  inside  protective 
armor  (e.g.,  see  figure  3).  Pliability  of  the  material  solution  is  important  so  as  not  to  compromise 
Soldier  safety  on  the  battlefield. 
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Figure  3.  Current  state-of-art  in  EEG  technology.  Left  image  shows 

configuration  and  labor-intensive  setup.  Right  image  illustrates  the 
potential  discomfort  and  temporary  skin  deformations  associated  with 
extended  use. 

As  an  alternative,  the  class  of  electronics  known  as  “stretchable  electronics”  holds  promise  for 
this  area.  It  is  distinguished  from  materials  known  as  “flexible  electronics”  by  three  fundamental 
differences:  (1)  an  order-of-magnitude  increase  in  the  spatial  scale  of  the  deformation,  (2)  a 
resulting  complex  nonlinear  shift  in  passive  electronic  behaviors  (resistance,  capacitance,  and 
inductance),  and  (3)  the  potentially  large  resulting  shift  in  signal  filtering  characteristics. 

Pliable,  stretchable  electronic  devices  have  only  recently  become  possible  due  to  advances  in 
materials,  including  those  made  by  ARL/WMRD  with  the  development  of  soft  mesoscale 
polymer  composites  capable  of  maintaining  low  levels  of  signal  attenuation  (less  than  3  dB) 
while  subjected  to  biaxial  hyperelastic  stretching  (see  figure  4).  The  data  was  acquired  with 
Agilent  E5061B  LF-RF  Impedance/Network  Analyzer  over  frequency  range  of  interest  for  EEG 
using  the  S21  series-through  method.  Sophisticated  and  unique  new  electromechanical 
characterization  techniques  were  recently  developed  by  ARL’s  VTD  that  allow  us,  for  the  first 
time,  to  make  accurate  measurements  of  the  influence  that  very  large  biaxial  strains,  frequency, 
temperature,  and  material  type  have  on  stretchable  electronic  signal  attenuation  and  filtering 
characteristics  (Slipher  et  al.,  2012). 
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Signal  Attenuation  For  ARL's  Stretchable  Conductor  (20% 
strain)  Compared  with  a  High-End  BNC  &  EEG  (0  strain) 


^Agilent  BNC  — Conductor  @  20%  Strain  — State-of-Art  EEG  Lead 


I 
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Figure  4.  Recent  experimental  data  acquired  by 

ARL/VTD  using  the  proposed  experimental 
techniques  on  initial  material  samples 
provided  by  ARL/WMRD.  Signal 
attenuation  for  ARL’s  stretched  conductor  at 
20%  strain  is  comparable  to  a  state-of-the-art 
EEG  lead. 


6.3.1  Program  Goals 

We  are  leveraging  these  recent  developments  ongoing  within  WMRD  and  VTD  to  meet 
emerging  mission  requirements  for  fieldable  biosensing  arrays  under  development  through 
HRED.  Our  goal  is  to  develop  spatially  distributed  and  tunable,  stretchable  conductors  for 
transmitting/filtering  signals  from  biological  sensors  to  data  acquisition  devices  (e.g.,  EKG  or 
EEG)  for  the  purpose  of  fielding  ruggedized  Soldier  biosensing  arrays.  There  are  three  principal 
scientific  challenges  that  we  will  address  in  this  research  program: 

1.  Materials  must  maintain  consistent  signal  attenuation  when  conductors  are  subjected  to 
deformations  such  as  those  that  can  be  anticipated  in  a  high-mobility  environment  instead 
of  increased  attenuation  as  is  more  typical. 

2.  The  appropriate  impedance  must  be  maintained  for  dry  contact  electrodes  interfacing  with 
human  skin  and  subjected  to  potentially  large  deformations  in  order  to  maximize  biosignal 
throughput  under  field  conditions. 

3.  Material-material  interfaces  will  have  to  be  designed  in  a  manner  that  establishes  an 
electrical  and  mechanical  connection  between  two  potentially  dissimilar  stretchable 
conductive  materials,  yet  minimizes  fatigue  failures  due  to  large  strain  gradients  at  the 
interface. 
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6.4  Program  Activity  During  FY12 

As  a  new  program  beginning  quarter  three  (Q3)  FY12,  the  primary  goal  for  this  year  was  to 
establish  material  development  methods  that  would  be  most  appropriate  for  EEG-type  devices, 
establish  a  research  plan,  and  secure  funding  for  future  efforts.  Description  of  development  and 
testing  efforts  follow.  Proposals  for  both  DRI  and  Director’s  Strategic  Initiative  funds  were 
submitted,  with  funding  granted  through  the  DRI  mechanism  to  provide  support  beginning 
FY13-FY15  (Slipher  et  al.,  2012b),  which  also  facilitated  planning  for  future  work. 

6.4.1  Pliable  Conductive  Polymer  Production 

6.4. 1.1  Replacement  EEG  Electrode  Tips 

Initial  effort  focused  on  developing  material  suitable  for  EEG  electrode  tips,  which  must  be 
pliable  to  provide  comfort  and  safety  for  long-term  use  on  the  scalp  but  also  maintain  consistent 
high  conductance  even  when  deformed  by  pressure  from  mounting  the  system  on  the  head. 
Electrode  tips  were  fabricated  by  dispersing  carbon  black  into  a 

poly-dimethyl  siloxane  (PDMS)  matrix.  The  PDMS  matrix  is  produced  through  reaction  of  a 
vinyl-terminated  PDMS  with  a  tetrafunctional  silane  cross-linker  in  the  presence  of  a  platinum 
catalyst.  This  chemistry  is  advantageous  for  these  applications  for  two  reasons:  (1)  there  are  no 
byproducts  from  the  cross-linking  reaction,  resulting  in  minimal  cure  shrinkage,  and  (2)  the  end- 
linked  functionality  provides  the  ability  to  systematically  tailor  the  mechanical  properties  over  a 
wide  range  of  modulus  to  provide  the  desired  response  and  offset  the  impact  of  the  carbon  black 
addition.  The  conductivity  of  the  composite  material  can  be  tuned  by  changing  the  carbon  black 
loading. 

Carbon  black  was  mixed  along  with  the  PDMS  network  precursors  using  a  Resodyn  resonant 
acoustic  mixer  at  a  power  of  60  dB  for  5  min.  The  mixture  was  then  poured  into  the  mold  and 
degassed  via  vacuum  to  remove  the  bubbles  incorporated  during  the  mixing  process.  The 
degassed  materials  were  placed  in  an  oven  at  72  °C  for  3  days  to  provide  complete  cure. 

6.4. 1.2  Stretchable  Wire  Conductors 

A  second  effort  addressed  the  substrate  for  wire  conductors,  for  which  a  carbon-black/DPMS 
matrix  would  not  be  suitable.  Conductors  were  fabricated  by  incorporating  nickel-coated  carbon 
fibers  (NCCF)  into  a  thermoplastic  elastomer  polymer  using  twin-screw  extrusion.  The 
thermoplastic  elastomer  matrix  provides  the  ability  to  melt  and  reform  the  material  using  heat  or 
dissolution  in  a  solvent.  The  material  was  compounded  in  a  DSM  conical  twin-screw  extruder 
(DSM  Resolve,  Geleen,  The  Netherlands)  at  200  °C  and  50  rpm  for  15  min  and  then  extruded 
through  a  3-mm  cylindrical  die.  The  extrudate  was  cut  into  an  approximately  150-mm  length  and 
melt-pressed  at  160  °C  and  90  kN  for  5  min  to  produce  a  uniform  sheet.  The  sheet  thickness  was 
controlled  by  placing  metallic  spacers  between  the  melt  press  platens.  This  allows  for  the 
production  of  large  conductive  elastomer  sheets.  For  example,  this  initial  study  utilized 
conductive  sheets  about  250  mm  x  250  mm  x  175  pm  and  exhibited  an  end-to-end  resistance  of 
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about  145  Q  (about  6  D/cm)  (figure  5).  When  subjected  to  an  equibiaxial  strain,  the  stretchable 
conductor  material  was  observed  to  initially  increase  conductivity.  This  increasing  conductivity 
with  strain  reached  a  peak  value  at  around  20%  area  strain,  with  a  measured  resistance  of  about 
3.4  Q/cm.  We  believe  that  by  tuning  the  properties  of  the  mesoscale  composite,  the  conductivity 
response  to  strain  can  be  tailored  to  give  a  desired  response  as  required  for  a  given  end  use.  For 
reference,  moisturized  EEG  leads  have  a  measured  conductance  of  about  2  Q/cm.  The  conductor 
material  was  cut  to  the  desired  geometry  using  a  commercial  laser  cutter.  Use  of  the  laser  cutting 
equipment  employed  to  date  limits  resolution  for  surface  geometries  to  greater  than  250-pm 
feature  size;  however,  this  is  within  the  range  necessary  to  mimic  current  EEG  electrical 
components. 


Figure  5.  A  conductive  elastomeric  sheet 

measuring  about  250  mm  x  250  mm 
x  175  (im  prior  to  laser  cutting. 

6.4.2  Production/Testing  of  Prototype  Devices 

Three  different  types  of  early  prototype  conductive  polymer  devices  were  produced  as  an  initial 
trial  application  for  the  existing  EEG  systems. 

6.4.2. 1  Replacement  EEG  Electrode  Tips 

Electrode  pads  were  made  to  replace  the  standard  saline-soaked  felt  pads  used  in  the  Emotiv 
Epoc  EEG  system.  Replacement  tips  were  made  using  carbon  black  substrate  mixed  with  PDMS, 
with  the  target  impedance  of  the  material  set  to  match  that  of  the  manufacturer’s  pads  (when 
soaked).  The  resulting  pads  were  fairly  soft  and  pliable.  Initial  testing  showed  that  although  the 
conductivity  was  approximately  equivalent,  it  was  extremely  sensitive  to  compression  and 
direction  of  surface  contact  such  that  slight  deflections  greatly  increased  the  signal-to-noise  ratio 
(SNR).  It  was  determined  that  this  approach  (carbon  black  with  DPMS)  is  not  suitable  for  this 
application  and  would  likely  not  address  challenge  no.  1  (page  17).  As  a  result,  alternative 
methods  will  be  explored  that  will  yield  better  consistency  across  contact  axes. 

6.4. 2. 2  Stretchable  Wire  Conductors 

A  set  of  EEG  electrode  wire  conductors  were  made  in  a  format  similar  to  that  of  commercially 
available  low-density  EEG  measurement  system  (Advanced  Brain  Monitoring  X-10)  and 
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embedded  in  a  stretchable  acrylic  coating.  Because  this  style  system  uses  open-cell  foam  gel- 
filled  pads  to  connect  to  the  wire  strip,  similar  exposed  ends  were  left  on  the  prototype  strip 
(figure  6).  This  prototype  yielded  positive  results,  with  conductance  remaining  broadly  within 
the  target  range  for  a  stretch  of  approximately  10%.  However,  while  this  provides  evidence  of 
successfully  addressing  challenge  no.  1,  it  was  evident  that  the  native  outer  coating  of  the  elastic 
conductor  poses  a  barrier  to  direct-contact  connection  and  will  require  future  effort  on  the 
interface  between  the  material  and  connections  (challenge  no.  2). 


Figure  6.  WMRD/VTD/HRED/Sensors  and  Electron 
Devices  Directorate  cooperative 
development  of  a  stretchable  EEG 
monitoring  device. 

6.4. 2. 3  Crimp-on  Connectors  With  Stretchable  Conductors 

A  crimp-on  style  metal  snap  was  attempted  as  an  interface  between  current  “dry”  electrodes  (in 
development  through  CaN  CTA)  and  stretchable  conductive  media.  Initial  attempts  suggest  that 
this  method  works  for  penetrating  the  outer  layer  of  the  media  and  makes  a  good  conductive 
connection.  Future  work  will  need  to  consider  whether  the  penetration  caused  by  the  crimp  will 
lead  to  tearing  of  the  material  with  repeated  use. 

6.5  Conclusions  and  Continuing  Efforts 

We  feel  that  the  flexible  conductive  materials  currently  under  development  in  ARL-VTD  and 
WMRD  hold  promise  for  the  application  to  EEG  or  similar  biosensing  modalities,  providing  a 
pathway  to  equipment  that  is  more  comfortable  and  safe  for  the  user  in  operational 
environments.  Based  on  the  initial  tests  described  here,  this  application  seems  feasible  for  the 
materials  and  methods  currently  in  development.  However,  considerable  effort  remains  before 
this  technology  can  be  used  in  a  reliable,  repeatable  manner  for  EEG.  For  example,  the  rapid 
change  in  conductance  that  occurs  with  compression  or  stretching  will  have  to  be  reduced  so  that 
small  fluctuations  do  not  dramatically  affect  the  measured  EEG  signal. 

Efforts  from  this  year  provided  guidance  on  which  material  methods  may,  and  may  not,  be 
suitable  for  future  investigation.  For  instance,  carbon -black- laced  DPMS  was  deemed  an 
unlikely  candidate,  while  the  NCCF  approach  appears  to  hold  promise.  Based  on  this,  continuing 
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and  near-future  efforts  on  materials  development  include  first  determining  the  critical  factors 
within  the  NCCF-based  approach  that  will  improve  performance  for  this  application.  Proposed 
methods  include  examining  the  factors  that  impact  the  ability  of  adjacent  particles  to  slide  in  a 
polymer  matrix,  exploiting  additives  to  decrease  the  surface  friction,  computational  modeling  of 
particle-particle  friction  in  a  polymer  matrix,  and  exploiting  particle -polymer  interaction  and 
polymer  microstructure  to  tailor  the  electrical  response  of  the  material.  Meanwhile,  we  will  be 
developing  processing  approaches  to  expand  integration  into  devices,  a  necessary  step  for 
building  future  prototype  devices  such  as  an  EEG  system.  Finally,  as  full-scale  prototype  device 
components  are  fabricated,  they  will  be  tested  in  conjunction  with  conventional  and  current- 
development  EEG  systems  to  assess  efficacy  against  standard  hard-conductor  approaches. 
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7.  Technologies  for  Evaluating  EEG  Capabilities 


7.1  Summary 

Contemporary  systems  used  for  electroencephalography  (EEG)  data  collection  are  extremely 
sensitive  to  influence  from  electromagnetic  interference  (EMI)  and  other  nonbiological  external 
sources  of  noise.  In  a  vehicle  environment,  this  may  arise  from  the  vehicle  engine  or  electrical 
system,  communications  equipment,  or  the  vibration  of  the  vehicle.  These  can  cause  artifacts  and 
distortions  that  make  the  data  difficult  or  in  some  cases  even  impossible  to  interpret.  As 
neuroscience  experiments  move  from  the  laboratory  to  the  field,  the  environment  becomes  less 
controlled  and  less  measurable,  introducing  novel  sources  of  noise  and  error  to  the  measuring 
devices.  However,  if  environmental  factors  are  known,  as  well  as  how  they  impact  the  recorded 
data,  then  many  sources  of  error  can  be  accounted  and  compensated  for.  One  way  to  distinguish 
the  externally  driven  noise  sources  from  naturally  occurring  brain  signals  is  through 
measurement  of  a  standard  model  within  the  various  target  environments.  At  the  moment,  no 
standard  model  exists  that  can  replicate  human  EEG-like  signals  in  a  medium  similar  to  the 
human  head,  forcing  most  system  designers  to  use  actual  humans  as  the  source  of  test  data. 
However,  this  approach  is  fraught  with  challenges,  such  as  within  or  across  person  variance  and 
lack  of  a  strong  known  signal  to  serve  as  a  ground-truth  comparison. 

To  address  this,  the  U.S.  Army  Research  Laboratory  (ARL)  has  initiated  a  Small  Business 
Innovative  Research  (SBIR)  project  (A  10-066,  “Neurological  Simulator  for  Applied  Data 
Collection”)  to  leverage  the  innovation  available  through  multiple  small  companies  motivated  by 
this  opportunity.  The  primary  goal  is  to  develop  a  model  human  head  that  will  replicate  the 
electrical  conductivity  of  the  human  head  and  provide  the  opportunity  to  create  signals  analogous 
to  human  brain  activity.  These  signals  can  then  be  measured  with  standard  EEG  equipment, 
allowing  the  opportunity  to  provide  a  “ground  truth”  signal.  Efforts  from  fiscal  year  2012 
(FY12),  performed  by  Physical  Optics  Corporation,  CFD  Research  Corporation,  and  Creare, 

Inc.,  focused  on  research  into  materials  and  potential  design  schemes,  as  well  as  development  of 
initial  prototype  “phantom”  devices  delivered  for  testing  to  ARL  (figure  3).  A  more  in-depth 
review  of  these  efforts  is  provided  in  the  appendix. 

This  EEG  phantom  will  provide  ARL  with  several  important  capabilities  for  developing  and 
evaluating  wearable  EEG  technologies  for  real-world  in-vehicle  environments.  First,  it  will  allow 
us  to  quantify  the  amount  of  artifact  and  noise  introduced  into  an  EEG  recording  in  a  real-world 
vehicle  system  by  using  EEG  to  record  a  known  signal  from  the  phantom  in  both  controlled 
laboratory  and  in-vehicle  conditions.  We  can  then  analyze  these  recordings  and  quantify  the 
difference,  allowing  ARL  to  develop  or  adapt  capabilities  to  address  these  differences. 


22 


Second,  the  phantom  will  allow  us  to  evaluate  new  and  existing  EEG  technologies  by  recording 
known  signals  from  of  the  phantom  using  a  variety  of  technologies  and  analyzing  the  resulting 
data  to  directly  compare  their  performance. 

7.2  Introduction 

Currently,  there  is  much  effort  underway  to  develop  physical  and  computational  tools  for 
measuring  brain  activity  outside  of  restricted  laboratory  environments.  Unfortunately,  there  is  no 
standard  way  for  testing  or  validating  these  tools,  due  in  large  part  to  a  lack  of  a  standard 
“model”  of  brain  activity  that  can  be  used  as  a  known-quantity  standard  for  comparison  of  either 
some  putative  tool  or  nonsignal  noise  emanating  from  the  environment.  The  work  described  here 
reflects  efforts  to  construct,  validate,  and  use  a  novel  “phantom”  device  for  EEG  equipment  and 
algorithms.  While  much  of  the  initial  design  and  fabrication  work  is  carried  out  via  the  U.S. 
Army  SBIR  process,  testing,  validation,  and  use  of  the  device  is  occurring  within  ARL  as  a  joint 
effort  between  ARL  and  SBIR  contractors. 

7.3  Background 

There  is  a  great  deal  of  current  work  in  the  human  performance  arena  that  focuses  on  the 
application  of  cutting-edge  neuroscience  tools  to  real-world  operational  environments.  This 
includes  the  integration  of  EEG  or  other  measures  of  nervous  system  state  with  other  sensor 
systems  during  specific  Soldier  activities.  However,  environmental  conditions  and  electrical 
noise  can  adversely  affect  the  collection  of  usable  physiological  and  neurological  measurements 
within  real-world  scenarios.  As  neuroscience  experiments  move  from  the  laboratory  to  the  field, 
the  environment  becomes  less  controllable  by  the  operator,  introducing  novel  sources  of  noise 
and  error  to  the  measuring  devices.  This  can  be  problematic,  but  if  environmental  factors  are 
known,  many  sources  of  error  can  be  accounted  and  compensated  for  in  experimental  design, 
recording  equipment  design,  or  artifact  modeling  during  online  and  offline  data  analysis. 
Likewise,  similar  techniques  can  be  used  for  testing  novel  recording  equipment  or  signal 
detection  software  tools. 

One  way  to  tease  apart  externally  driven  noise  sources  from  naturally  occurring  and  expected 
brain  signal  is  through  measurement  of  a  standard  or  known-quantity  model  within  the  various 
target  environments.  At  the  moment,  however,  no  standard  model  exists  that  can  replicate  human 
EEG-like  signals  within  a  physical  structure  that  is  similar  in  size,  shape,  and  consistency  to  the 
human  head.  While  there  currently  are  a  number  of  computational  models  outlining  the 
conductive  properties  of  brain  matter,  skull,  and  skin,  no  physical  models  are  commercially 
available  that  also  have  analogous  conductive  properties.  Likewise,  the  few  physical,  conductive 
models  or  “phantoms”  discussed  in  academic  literature  (e.g.,  Gavit  et  al.,  2001;  Leahy  et  al., 
1998;  Looi  and  Chen,  2005;  Miller  et  al.,  2000)  or  made  in  a  handful  of  laboratories  either  have 
not  been  created  in  an  appropriate  shape  for  use  with  complete  conventional  EEG  caps  or  soldier 
headgear.  Being  structured  as  such  would  enable  use  as  a  benchmark  for  typical  off-the-shelf 
cap-based  EEG  systems. 
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To  address  this  deficit,  an  innovative  system  is  needed  that  will  replicate  signals  analogous  in 
size  and  scope  to  human  brain  activity,  which  can  in  turn  be  measured  with  standard  EEG 
equipment,  allowing  the  opportunity  to  provide  a  true  “ground  truth”  signal.  In  order  to  properly 
simulate  the  perturbation  in  measurement  of  real  signals  generated  by  human  participants  within 
varied  environments,  the  model  will  have  to  be  physically/conductively  analogous  to  the  human 
skull,  scalp,  and  internal  structure  to  functionally  replace  a  human  during  experimental  setup  and 
testing  and  contain  multiple  sources  for  creating  EEG-like  signals  from  distinct  interior  points. 

A  phantom  device  such  as  described  above  has  use  in  a  wide  range  of  applications  (from 
validating  lab  or  medical  grade  EEG  systems  to  testing  unknown  environments  or  training  new 
personnel);  as  a  result,  there  is  a  high  potential  for  marketability  of  the  target  or  related  devices. 
As  such,  much  of  the  initial  design,  fabrication,  and  first-level  testing  is  occurring  through 
contractors  participating  in  the  U.S.  Army  SBIR  award  process  (Army  Topic  #A30-066). 

7.3.1  Program  Goals 

The  goal  of  this  project  is  the  development,  validation,  and  use  of  a  phantom  device  to  be  used 
for  EEG  and/or  related  equipment  based  on  measuring  voltage  on  the  human  scalp.  Our  vision  of 
the  device  is  that  it  will  be  conductively  analogous  to  the  human  brain,  skull,  and  scalp,  with 
different  physical  layers,  and  contain  multiple  (16-32)  embedded  electrical  sources  that  can  be 
used  to  simulate  signals  that  are  consistent  with  human  brain  activity.  Finally,  because  the  model 
will  be  used  as  a  test  fixture  within  a  wide  variety  of  environments  and  varied  conditions,  it 
should  be  fairly  durable,  rugged,  and  portable,  requiring  minimal  maintenance,  and  be  able  to 
survive  use  within  military  operational  environments. 

The  end  goal  is  to  develop  a  device  that  can  be  used  not  only  for  examining  and  eliminating 
sources  of  error  and  noise  within  applied-research  project  environments  but  also  as  a  standard 
tool  for  comparing  and  validating  new  EEG  technology  and  algorithms  within  these  or  other 
environments  as  it  becomes  available.  That  is,  it  will  serve  as  a  gold  standard  for  future  materials 
testing. 

The  overall  program  is  anticipated  to  include  three  primary  areas: 

•  Fabrication  of  a  complete  head-shaped  phantom  system.  This  effort  will  involve  the  design 
and  building  of  a  physical  system  and  will  inevitably  include  building  several  prototype 
devices  and/or  subcomponents,  as  necessary,  to  determine  the  appropriate  properties, 
materials,  manufacturing  methods,  and  testing  procedures  for  building  the  final  device  that 
meets  the  necessary  requirements.  We  anticipate  the  majority  of  this  work  to  occur  external 
to  ARL’s  Human  Research  and  Engineering  Directorate  (HRED)  through  the  SBIR  or 
other  external  processes,  with  recurring  interaction  and  feedback  from  ARL  personnel. 

•  Validation  of  the  device  and  initial  testing.  Once  prototype  devices  are  developed,  they 
must  be  validated  as  indeed  having  the  target  qualities  (e.g.,  realistic  conductance, 
durability,  etc.)  and  proven  to  be  usable  for  ARL’s  intended  purposes.  This  effort  is 
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expected  to  be  performed  in  conjunction  with  an  SBIR  contract  as  well  as  some  testing 
within  ARL,  with  recurring  feedback  between  the  two  groups. 

•  Use  of  the  device  as  an  exemplar  test  fixture.  As  initial  devices  are  fabricated  and 

validated,  they  will  be  used  as  test  fixtures  for  various  ARL  applications,  such  as  assessing 
ambient  (EMI,  testing  new  EEG  hardware,  or  assessing  the  efficacy  of  novel  signal 
detection  schemes.  This  testing  will  serve  two  purposes:  first,  to  highlight  the  utility  of  the 
device  as  a  useful  scientific  and  engineering  tool  and  second,  to  provide  opportunity  for 
feedback  to  fabricators  for  design  improvements  to  ensure  maximum  efficacy  of  the 
device.  The  majority  of  this  effort  will  be  performed  by  ARL  and  collaborating  partners. 

7.4  Previous  Program  Activity 

7.4.1  SBIR  Overview 

Effort  on  this  project  began  in  the  third  quarter  (Q3)  of  FY 10  with  the  solicitation  of  proposals 
through  the  Army  SBIR.  The  primary  task  of  the  project  was  to  outline  designs  and  provide 
evidence  of  capability  to  fabricate  a  physical  phantom  head-shaped  device  used  for  the  purposes 
described  previously.  Three  individual  contracts  were  selected  from  different  groups  working 
independently — CFD  Research  Corporation  (CFDRC),  Physical  Optics  Corporation  (POC),  and 
Creare,  Inc.,  with  efforts  beginning  Q2  FY  11.  All  three  proposed  to  deliver  a  physical  prototype 
device  as  an  example  by  the  end  of  phase  I,  but  each  company  placed  an  emphasis  on  different 
aspects  of  the  project  and  proposed  system.  A  brief  review  of  the  efforts  and  device  delivered  by 
each  group  and  ending  in  Q4  FY  1 1  is  outlined  in  the  following  sections.  Detailed  information 
can  be  found  in  the  SBIR  annual  reports. 

7.4.2  CFDRC 

This  project  emphasized  using  computational  modeling  to  derive  the  appropriate  structure  and 
electrical  properties  for  the  phantom  device;  the  computational  model  on  which  the  physical 
device  was  based  was  delivered  with  the  prototype.  The  delivered  model  uses  a  gel-filled  cavity 
as  the  “brain”  inside  a  plastic  mannequin  head,  with  fixed  silver  epoxy  channels  to  make  contact 
with  the  surface  (figure  7,  A).  As  delivered,  the  device  has  an  embedded  USB-programmable 
function  generator  connected  to  the  conductive  channels. 

7.4.3  POC 

POC  focused  on  the  material  properties  and  manufacturing  process  to  create  realistic 
conductivity  for  differing  brain/skull/scalp  layers.  Their  approach  was  a  carbon  nanotube 
polymer  mixture,  where  the  mixture  could  be  altered  to  reach  the  target  conductance  for  each 
layer.  They  also  focused  on  identifying  a  relatively  cost-effective  manufacturing  process.  The 
delivered  prototype  was  a  sphere  of  hard  nanotube  based  polymer,  with  two  embedded  electrodes 
connected  to  a  built-in  function  generator  (figure  7.  B). 
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7.4.4  Creare,  Inc. 

This  contractor,  who  worked  closely  with  faculty  at  Dartmouth  College,  was  concerned  with 
having  both  conductive  and  physical  realism  within  their  prototype  model.  They  completed  a 
series  of  tests  using  differing  levels  of  an  elastomer  mixed  with  carbon  fiber  to  reach  the  target 
realistic  conductance,  and  fabricated  a  head-shaped  prototype  based  on  the  magnetic  resonance 
imaging  (MRI)  of  a  human  (figure  7,  C).  It  includes  three  separate  layers  (brain,  skull,  and  scalp) 
and  has  16  embedded  electrodes  (although  several  are  nonfunctional)  and  an  external  function 
generator,  making  it  fairly  universal  for  different  types  of  testing. 


Figure  7.  Prototype  phantom  devices  delivered  by  (A)  CFDRC,  (B)  Physical  Optics  Corporation,  and  (C)  Creare,  Inc. 

7.5  Program  Activity  During  FY 12 
7.5.1  Fabrication  and  Validation  of  Devices 

At  the  beginning  of  FY12,  Creare,  Inc.,  was  selected  as  the  primary  performer  for  the  SBIR 
phase  II  portion  of  this  effort.  During  the  phase  I  option  period,  which  continued  through  Q1 
FY12,  they  continued  examination  into  more  reproducible  long-term  viable  manufacturing 
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processes  and  settled  upon  a  combined  three-dimensional  (3-D)  printing  and  injection-molding 
approach  as  a  potential  method  for  future  efforts. 

However,  due  to  a  long-term  delay  in  external  contracting,  Creare  has  not  yet  been  awarded  the 
contract  for  phase  II  work.  As  a  result,  no  effort  of  note  has  occurred  in  these  areas;  all  work 
within  ARL  has  been  based  on  the  early  prototypes  delivered  during  phase  I  SBIR  efforts  in 
FY11. 

7.5.2  Use  of  Device  as  an  Exemplar  Test  Fixture 

During  FY 12,  the  prototype  devices  were  used  in  a  number  of  applications  at  ARL,  highlighting 
their  utility  as  a  test  fixture  as  well  as  a  means  of  identifying  a  number  of  shortfalls  in  the  current 
design.  The  Creare,  Inc.  prototype  was  used  exclusively.  Some  examples  are: 

•  Testing  timing  accuracy  in  multiple  commercial  EEG  systems.  EEG  is  often  collected  as  a 
response  to  particular  events,  in  which  case  having  precise  time-marking  within  the  system 
is  critical.  We  used  the  phantom  head  to  create  a  known  time-locked  signal  measured  by 
four  different  commercially  available  EEG  systems  (e.g.,  figure  8).  The  variance  in 
accuracy  with  this  data  was  then  compared,  yielding  the  systems  that  are  not  reliable  for 
laboratory  use.  This  data  effort  has  been  described  in  Hairston  (2012). 


Figure  8.  EEG  Phantom  used  as  a 
fixture  for  testing  timing 
response  reliability  with  a 
standardized  input  signal. 

•  Validation  of  a  timing  correction  algorithm.  Given  the  high  degree  of  variance  and  drift 
observed  in  actual  timing  for  some  EEG  systems  used  at  ARL,  it  was  necessary  to  develop 
a  correction  algorithm  to  compensate  for  these  problems.  Once  developed,  the  phantom 
device  was  used  to  provide  a  known-standard  signal  to  validate  the  efficacy  of  the 
algorithm  by  comparing  the  shape  of  the  recorded  waveform  to  the  known  signal,  both 
with  and  without  correction.  This  effort  has  been  described  in  Hairston  (2012)  and  multiple 
conference  presentations  including  Hairston  and  Ries  (201 1)  and  Hairston  et  al.  (2012). 
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•  Testing  timing  accuracy  in  a  prototype  system.  Collaborators  at  Taiwan’s  National  Chiao 
Tung  University  (NCTU),  part  of  the  Cognition  and  Neuroergonomics  Collaboative 
Technical  Alliance  (CaN  CTA)  program,  have  developed  new  versions  of  the  Mindo  EEG 
headset  and  related  software  that  are  unique  approaches  to  the  field.  We  used  the  device  to 
determine  the  timing  integration  consistency  of  the  new  system,  and  attempted  to  use  it  as  a 
standard  signal  generator.  Results  showed  that  the  prototype  NCTU  system  has 
considerable  timing  errors  that  must  be  addressed  and  that  the  current  design  of  the 
phantom  is  not  compatible  with  the  current  NCTU  hardware  approach.  We  provided  this 
feedback  to  CaN  CTA  collaborators. 

•  Assessing  variance  in  EEG  electrode  placement  across  systems.  A  common  problem  with 
different  EEG  systems  is  that  the  actual  electrode  locations  on  the  scalp  may  not  be 
consistent  across  hardware  approaches.  Because  the  prototype  phantom  is  modeled  from  a 
real  human  head,  it  was  used  as  a  standard  to  compare  how  well  each  of  four  different 
systems  actually  matched  the  standard  “10-20”  locations  and  estimate  how  error  would  be 
introduced  by  different  head  sizes.  We  discovered  that  the  prototype  model  is  not  well 
suited  for  this  use  (due  to  surface  irregularities)  and  gave  feedback  to  the  SBIR  contractors 
at  Creare. 

7.6  Conclusions  and  Continuing  Efforts 

Based  on  experiences  thus  far,  the  phantom  EEG  devices  under  development  appear  to  be  of 
practical  use  for  ARL  and  likely  the  neuroscience  research  community  as  a  whole.  However, 
much  work  remains  before  a  device  is  available  that  fully  suits  the  needs  of  ARL  and  could 
feasibye  be  marketed  as  a  commercial  product  by  the  contractor.  Overcoming  the  challenges  that 
have  been  identified  thus  far  will  be  the  focus  of  effort  over  the  coming  years.  Near-future  effort 
in  fabrication  will  focus  on  materials  and  construction  processes  that  yield  fairly  uniform 
conductivity  and  proper  conductance  at  the  outer  skin  layer.  Additionally,  SBIR  contractors  will 
be  investigating  new  materials  for  the  inner  shell  that  will  enhance  stability  as  well  as 
consistency  in  production.  Meanwhile,  efforts  within  ARL  will  focus  on  continued  testing  and 
exemplar-use  cases  of  devices,  with  constant  interaction  between  personnel  and  contract 
performers. 
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8.  Evaluation  of  Existing  Driver  Performance  Prediction  Technologies 


8.1  Summary 

8.1.1  Introduction 

The  human  operator  plays  a  critical  role  in  vehicle  survivability.  For  example,  in  civilian  and 
industrial  situations,  analyses  show  that  human  error  is  at  least  partly  to  blame  in  over  90%  of 
vehicle  crashes  (Treat  et  al.,  1977).  Specifically,  fatigue  and  drowsiness  are  among  the  primary 
contributors  to  vehicular  crashes,  which  have  been  found  to  account  for  approximately  one-third 
of  U.S.  military  fatalities  annually  (Krahl  et  al.,  2010).  Over  the  past  decade,  there  have  been 
substantial  worldwide  research  efforts  toward  developing  systems  designed  to  detect  and 
mitigate  driver  fatigue  (Kerick  et  al.,  in  press).  As  a  result,  ARL  is  researching  the  development 
of  neurotechnologies  designed  to  enhance  survivability  through  mitigating  fatigue-based  vehicle 
operator  error,  which  is  the  focus  of  this  collective  effort. 

For  years,  researchers  and  clinicians  have  viewed  directly  measuring  electrical  activity  from  a 
person’s  brain  through  electroencephalography  (EEG)  as  the  gold  standard  in  drowsiness  and 
sleep  detection.  Recent  breakthroughs  in  neuroimaging  technology  have  made  using  EEG  in 
vehicles  feasible,  opening  the  possibility  for  improving  driver  fatigue  detection  technologies., 
Lin  et  al.  (2005a)  attempted  to  extend  these  ideas  beyond  simply  detecting  the  drowsiness  of  the 
driver  to  using  EEG  to  predict  driving  performance  directly.  The  results  showed  considerable 
promise  for  systems  designed  to  forecast  driver  error  based  solely  on  EEG  recordings.  While 
their  algorithm  was  largely  effective,  the  driving  scenario  they  used  was  highly  simplified  when 
compared  to  military-relevant  secure  mobility  tasks.  This  current  effort  focuses  on  replicating 
Lin  et  al.’s  (2005a)  algorithm  to  identify  areas  for  enhancement  and  improvement.  It  is  part  of  a 
broader  effort  to  extend  this  algorithm  to  more  naturalistic  driving  tasks  as  well  as  to  other 
military-relevant  situations  where  vigilance  and  alertness  are  imperative. 

8.1.2  Methods 

During  a  single  45-min  driving  session,  14  subjects  drove  down  a  long  straight  highway. 
Subjects  controlled  steering  and  speed  and  had  to  observe  posted  speed  limits.  Approximately 
every  10  s,  a  strong  lateral  force  pushed  the  vehicle  to  the  left  or  right.  Driver  performance  was 
defined  as  “lane  deviation,”  the  distance  from  the  center  of  the  vehicle  to  the  center  of  the 
driving  lane.  Lane  deviation  was  measured  continuously  at  approximately  100  Hz.  Throughout 
the  driving  session,  64-channel  EEG  signals  were  recorded  at  2048  Hz.  EEG  signals  were  band¬ 
pass  filtered  to  the  1-  to  50-Hz  range  and  then  used  to  compute  the  EEG  log  power-spectrum  for 
1-  to  40-Hz  frequency  bands.  Both  EEG  spectra  and  lane  deviation  were  smoothed  offline  using 
a  retrospective  90-s  moving  average  window,  iterating  at  2-s  steps. 


30 


The  EEG  and  lane  deviation  data  were  divided  into  two  blocks:  a  training  block  (first  15  min) 
used  to  train  a  regression  model  and  a  testing  block  (last  30  min)  used  to  evaluate  model 
performance.  Correlation  of  the  smoothed  EEG  power  spectra  and  lane  deviation  training  data 
were  calculated  for  each  EEG  channel  and  frequency  band  between  1  and  50  Hz.  The  two 
channels  with  highest  average  correlation  coefficients  were  selected  for  principal  component 
analysis  (PCA).  The  resulting  PCA  scores  of  the  first  50  eigenvectors  were  used  to  generate  the 
linear  regression  for  lane  deviation.  This  model  was  used  to  predict  the  lane  deviation  for  the 
remainder  of  the  session  based  solely  on  the  power-spectra  of  the  PCA  scores  of  testing  data. 

Lin’s  “global”  approach  trains  and  tests  the  algorithm  over  a  relatively  long  period  of  time.  To 
compare  this  approach  to  one  focusing  on  shorter  time  frames,  we  also  developed  a  “local”  form 
of  the  algorithm  that  trains  on  only  the  previous  60  s  of  power-spectra  and  lane  deviation  and 
predicts  only  the  next  20  s  iteratively  across  the  entire  session.  This  approach  may  provide 
greater  sensitivity  to  small  or  transient  changes  in  the  connection  between  EEG  signals  and 
driver  performance;  however,  this  added  sensitivity  may  in  turn  yield  erratic  predictions  for 
testing  data. 

8.1.3  Results 

In  general,  the  training  of  the  global  and  local  algorithms  yielded  similar  performance  to  that 
reported  by  Lin  et  al.  (2005a)  on  the  training  data  session,  yielding  an  average  correlation  of 
0.91/0.94  for  global/local  approaches,  respectively.  However,  performance  of  both  global  and 
local  regressions  significantly  declined  when  applied  to  the  testing  data,  yielding  respective 
average  correlations  of  0.05/0.61.  While  the  performance  of  the  local  regression  declined 
significantly  less  than  the  global  approach,  its  predictions  were  generally  much  noisier  and  at 
times  resulted  in  drastic  changes  in  the  predicted  driver  performance. 

8.1.4  Discussion 

From  these  results,  it  appears  that  the  increased  complexity  of  a  naturalistic  driving  task  poses  a 
significant  challenge  to  EEG-based  driver  performance  prediction.  Nonetheless,  we  do  see  a 
large  potential  for  improving  upon  these  linear  regression  algorithms.  To  that  end,  we  are 
currently  investigating  several  avenues  that  we  have  identified  as  potential  means  to  advance  the 
concept  of  EEG-based  prediction  of  driver  performance: 

•  Adjusting  the  width  or  iteration  rate  of  the  moving  average  window  may  improve 
performance  by  optimizing  how  much  data  are  combined  into  a  given  data  point  used  by 
the  regression. 

•  Combining  the  global  and  local  approaches  may  offer  improved  performance  over  either 
approach  individually.  Each  approach,  global  and  local,  offer  distinct  advantages  that  if 
integrated  appropriately  (e.g.,  on  the  basis  of  their  covariances)  could  provide  a  more 
accurate  and  more  stable  prediction  of  driver  performance. 
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•  Enhanced  behavioral  metrics  (e.g.,  eye-tracking/blinking,  steering  control,  time-to-contact 
of  lane  boundary)  are  also  being  considered.  Like  lane  deviation,  these  measures  have  a 
solid  link  to  driver  fatigue,  but  may  also  be  tied  to  EEG  signals  to  improve  predictive 
performance. 

•  To  explore  improvements  to  the  regression  algorithm  itself,  Support  Vector  Regression 
(SVR)  is  also  being  used  to  predict  driver  performance.  SVR  has  proven  to  be  a  highly 
effective  and  easily  implemented  machine-learning  approach,  and  is  likely  to  be  more 
robust  to  variability  in  the  driver  performance  metric  associated  with  more  complex  driving 
conditions. 

•  Network  connectivity  measures  are  also  being  explored  to  assess  whether  driver 
performance  can  be  predicted  from  a  combination  of  EEG  electrode  sites  better  than  if  the 
channels  are  considered  independently,  as  in  Lin’s  approach. 

•  In  addition  to  EEG  signals,  we  have  access  to  multiple  other  measures  that  may  be  used  to 
predict  driver  fatigue  and  performance.  Measures  such  as  eye-tracking  and  blinking  as  well 
as  aspects  of  steering  wheel  variability  and  reversals  have  been  used  to  predict  driver 
fatigue.  Thus,  combining  these  approaches  with  an  EEG-based  estimator  of  driver 
state/performance  may  yield  a  system  that  out-performs  any  single-measure -based 
prediction. 

•  A  multitier  classification  of  driver  performance  may  provide  the  appropriate  level  of 
insight  into  driver  fatigue  while  also  improving  predictive  accuracy.  To  investigate  this 
potential,  Support  Vector  Classification  (SVC)  based  approaches  to  predicting  driver 
performance  from  EEG  signals  are  also  being  explored. 

8.1.5  Conclusion 

Given  the  encouraging  results  of  the  Lin  et  al.  approach  described  here,  we  believe  that  these 
proposed  extensions  are  likely  to  facilitate  the  expansion  of  their  basic  method  to  increasingly 
more  complex  driving  environments.  It  is  our  further  hope  that  the  resulting  algorithm  will  have 
applications  beyond  the  vehicle  and  be  capable  of  reliably  predicting  lapses  in  performance  in 
any  task  that  requires  an  alert  and  attentive  soldier.  See  the  appendix  for  additional  detail. 

8.2  Introduction 

Fatigue  and  drowsiness  are  among  the  primary  contributors  to  vehicular  accidents,  being 
estimated  to  have  contributed  to  over  90%  of  all  accidents  (Treat  et  al.,  1977).  As  a  result,  the 
prevention  of  these  accidents  has  become  a  major  focus  of  driver  safety  research  (for  a  summary 
of  the  major  findings  in  this  area,  see  the  review  on  fatigue  technology  from  Kerick  et  al.  (in 
press).  This  is  a  particularly  relevant  issue  for  today's  military,  as  it  has  been  recently  estimated 
that  vehicle  accidents  account  for  approximately  one-third  of  military  fatalities  (Krahl  et  al., 
2010).  Unfortunately,  many  factors  contribute  to  drowsiness  or  fatigue  while  driving,  including 
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long  working  hours,  lack  of  sleep,  use  of  medication,  and  the  nature  of  the  task — for  example, 
long-distance  convoy  transport.  In  fact,  a  2005  poll  conducted  by  the  National  Sleep  Foundation 
found  that  about  60%  of  adult  drivers  admitted  to  getting  behind  wheel  in  a  drowsy  or  fatigued 
state  (National  Sleep  Foundation,  2005).  In  light  of  this,  there  has  been  a  significant  push  to 
develop  a  system  capable  of  detecting  driver  drowsiness  and  initiate  mitigating  strategies  to 
remedy  the  situation. 

Toward  this  end,  many  systems  have  been  designed  to  detect  driver  fatigue,  several  of  which 
display  relatively  good  accuracy.  Typically,  these  systems  have  relied  on  vehicle-mounted 
sensors  that  correlate  certain  behaviors,  such  as  driver  posture  or  eye -blinking  characteristics 
(Smith  et  al.,  2000;  Perez  et  al.,  2001;  Popieul  et  al.,  2003).  In  addition  to  these  behavioral 
metrics,  a  number  of  recent  studies  have  attempted  to  use  EEG  signals  to  predict  the  onset  of 
driver  fatigue  and  drowsiness.  If  successful,  an  EEG-based  system  may  offer  important 
advantages  over  behavioral  measures,  such  as  the  ability  to  predict  drowsiness  well  in  advance  of 
its  behavioral  manifestations,  and  may  be  able  to  quantify  the  level  of  drowsiness  of  a  driver  as  it 
evolves.  Intriguingly  and  encouragingly,  a  number  of  these  studies  have  found  significant 
correlations  between  neural  signals  and  driver  fatigue  (for  a  review,  see  Lai  and  Craig,  2001) 
despite  differing  widely  in  their  approach,  and  have  led  to  the  development  of  several  algorithms 
capable  of  classifying  periods  of  driver  fatigue. 

Lin  et  al.  (2005a)  have  attempted  to  not  only  determine  fatigue  onset  but  directly  predict  driver 
behavior  based  solely  on  neural  activity  recorded  while  driving.  To  do  this,  they  have  applied  a 
linear  regression  algorithm  to  EEG  signals  to  predict  subjects  ability  to  compensate  for  vehicle 
perturbations  with  impressive  results.  However,  the  driving  simulation  used  in  their  task  was 
very  simplistic,  and  thus  it  remains  unclear  if  such  this  regression  model  would  perform 
adequately  in  more  complex  driving  tasks  or  across  a  wider  range  of  subjects.  To  begin  to 
address  this  question,  we  sought  to  reimplement  the  approach  detailed  in  Lin  et  al.  (2005a)  to  a 
more  realistic  driving  simulation  in  which  a  subject  must  maintain  continuous  active  control  of 
the  vehicle’s  position  and  speed  while  responding  to  road  signs  and  lateral  perturbations  of  the 
vehicle. 

8.3  Methods 

8.3.1  Subjects 

A  total  of  14  subjects  (aged  from  20  to  40  years)  participated  in  the  SVR-based  highway  driving 
experiments.  Each  subject  was  briefed  on  the  experimental  equipment  and  procedures  and  signed 
an  informed  consent  form.  Protocol  and  forms  were  approved  by  the  Army  Research  Laboratory 
human-use  board  (Project  #  ARL  10-051). 

8.3.2  Experimental  Protocol 

Subjects  completed  two  separate  driving  sessions:  the  first  baseline  session  lasted  15  min  and  the 
second  experimental  session  consisted  of  45  min  of  continuous  driving.  Before  each  session, 
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subjects  provided  an  estimate  of  their  fatigue  level  via  the  Karolinska  Sleepiness  Scale  (KSS) 
(Akerstedt  and  Gillberg,  1990).  Additionally,  subjects  were  asked  to  verbally  report  their  fatigue 
score  on  this  scale  every  15  min  during  the  second  experimental  session. 

Subjects  were  instructed  to  keep  their  vehicle  as  close  to  the  center  of  the  right-hand  lane  as 
possible.  After  the  simulator  determined  the  subjects  had  been  within  the  appropriate  lane  for 
8  to  1 1  s,  a  lateral  perturbation  was  applied  to  the  vehicle,  causing  it  to  veer  off  course  and  if 
unabated,  out  of  the  lane.  The  perturbation  continued  to  push  the  vehicle  until  the  driver  made  a 
corrective  steering  adjustment,  defined  as  a  steering  wheel  deflection  of  one  degree  in  the 
opposite  direction  of  the  perturbation,  at  which  point  the  perturbation  ceased  and  the  subject  was 
to  return  the  vehicle  to  center  of  the  driving  lane.  If  the  subject  did  not  perform  a  corrective 
steering  adjustment,  the  perturbation  would  ramp  down  automatically  after  approximately  3  s; 
however,  it  was  left  to  the  subject  to  correct  the  vehicle’s  heading  and  position.  In  addition  to 
maintaining  control  of  the  vehicle's  direction,  drivers  also  maintained  appropriate  speed  for  the 
vehicle  during  the  testing  session.  Subjects  controlled  their  speed  via  accelerator  and  brake 
pedals  and  were  instructed  to  obey  posted  speed  limit  signs  that  appeared  during  the  driving 
session.  For  the  majority  of  the  session,  the  speed  limit  was  45  mph.  At  three  different  points 
during  the  testing  session,  the  posted  speed  limit  was  reduced  to  25  mph. 

8.3.3  Data  Collection 

Vehicle  status  and  performance  metrics.  Vehicle  status,  sampled  at  approximately  100  Hz,  was 
monitored  throughout  each  session.  Several  relevant  measures  were  recorded,  with  the  present 
analysis  focusing  primarily  on  lane  deviation — the  distance  between  the  center  of  the  vehicle  and 
the  center  of  the  driving  lane. 

Electroencephalography.  EEG  signals  were  collected  using  a  64-channel  Biosemi  EEG  system, 
sampled  at  2048  Hz.  Electrode  offset  was  configured  to  be  less  than  25  mV  before  proceeding 
with  data  collection. 

Eye  tracking.  Eye  position  was  monitored  throughout  the  experiment  using  SensoMotoric 
Instruments  (SMI)  eye-tracker,  sampled  at  256  Hz. 

8.3.4  Data  Analysis 

Vehicle,  EEG,  and  eye-tracker  data  were  collected  simultaneously  throughout  the  experiment. 
Specific  event  markers  were  embedded  within  each  data  structure  and  used  to  align  the  data  in 
time  and  remove  any  drift  or  jitter  in  the  time  series  of  each  data  stream. 

Lane  deviation  was  used  as  the  primary  metric  of  driver  performance  over  time,  as  has  been  used 
in  similar  fatigue  prediction  studies  (Lin  et  al.,  2005a).  Lane  deviation  values  over  the  entire 
session  were  smoothed  using  a  90-s  moving  average  filter  with  2-s  increments.  This  was 
advantageous,  as  use  it  yields  a  driver  performance  metric  that  illustrates  longer  scale 
fluctuations  in  performance,  rather  than  one  which  is  more  susceptible  to  variability  associated 
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with  the  task  (e.g.,  the  perturbations  that  artificially  increase  lane  deviation).  A  similar  process 
was  performed  on  the  steering  wheel  orientation  collected  during  each  driving  session. 

The  64-channel  EEG  data  was  collected  at  2048  Hz  and  then  down-sampled  to  256  Hz.  Using 
the  embedded  timing  pulses  and  event  signals,  the  EEG  time  series  was  synchronized  with  the 
vehicle  status  and  driver  performance  data.  Following  this,  the  data  was  band-pass  filtered  to 
remove  signals  greater  than  50  Hz  and  less  than  1  Hz.  The  data  for  each  channel  was  then 
converted  into  its  power  spectrum  using  a  750-point  Hanning  window  with  250-point  overlap. 
Each  EEG  channel  and  frequency  power  estimate  of  the  1-40  Hz  bands  was  then  passed  through 
the  same  90-s  moving  average  filter  used  to  smooth  the  lane  deviation  data,  preserving  the 
temporal  alignment  of  the  two  data  streams.  The  smoothed  lane  deviation  data  was  subsequently 
used  as  a  metric  of  driver  performance  and  used  to  train  a  linear  regression  model  with  the 
simultaneously  recorded  EEG  data  used  as  a  predictor  of  the  driver  performance  metrics. 

Performance  prediction.  The  data  from  the  experimental  session  was  split  into  two  subsections, 
the  first  15  min  of  data  reserved  for  training  and  the  last  30  min  of  data  for  testing  the  driver 
performance  prediction  regression.  The  first  step  in  this  was  to  use  the  training  data  to  identify 
the  frequency  bands  in  which  EEG  channels  varied  most  consistently  with  the  measures  of  driver 
performance.  A  correlation  analysis  was  performed  between  lane  deviation  data  and  power 
estimate  for  each  frequency  band  for  each  EEG  channel.  The  top  two  channels  that  were  on 
average  most  positively  correlated  with  driver  performance  were  selected  for  further  analysis. 

PC  A  was  then  performed  on  the  power  spectral  bands,  and  the  resulting  eigenvalues  were  used  to 
reduce  the  dimensionality  of  the  EEG  feature  space  from  100  to  50  dimensions. 

The  data  from  the  training  portion  of  the  driving  session  was  projected  into  principal  component 
space  and  the  first  50  eigenvectors  were  used  as  input  variables  to  train  a  50-order  linear 
regression.  To  assess  the  quality  of  the  fit  of  the  regression  model,  the  same  EEG  data  used  to 
train  the  regression  model  was  input  into  the  algorithm  again  to  predict  lane  deviation  during  the 
training  session.  This  output  was  compared  to  the  actual  levels  of  driver  performance. 

To  evaluate  the  algorithm’s  ability  to  predict  driver  performance,  the  regression  coefficients 
calculated  with  the  training  data  was  then  applied  to  PCA  projections  of  the  EEG  data  reserved 
as  the  testing  portion  of  the  session.  The  predicted  values  of  lane  deviation  generated  by  the 
model  were  compared  to  the  measured  driver  performance  for  each  epoch  to  characterize  the 
predictive  accuracy  of  the  approach. 

Loccd  vs.  globed  regression.  The  described  algorithm  essentially  adapts  the  Lin  et  al.  (2005a) 
algorithm  to  our  in-house  driver  simulator.  In  its  original  form,  the  algorithm  generates  a 
prediction  over  a  long  span  of  time  and  is  also  trained  over  a  long  time  period;  thus,  we  can 
consider  this  a  global  regression  approach.  To  evaluate  the  potential  of  using  only  the  most 
recent  driving  behavior  and  brain  activity,  we  developed  a  second  algorithm  that  uses 
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the  same  regression  approach  applied  to  a  much  narrower  space  of  time.  Using  only  the 
preceding  60  s  of  data  for  training,  the  algorithm  was  used  to  predict  the  next  20  s  of  driver 
performance.  This  prediction  algorithm  can  then  be  applied  iteratively  across  the  entire  driving 
session  to  provide  a  local  prediction. 

8.4  Results  and  Discussion 

Global  regression.  Figure  9  illustrates  the  measured  lane  deviation  (black  line)  as  the  predicted 
values  (blue  line)  produced  by  using  the  global  form  of  the  Lin  et  al.  (2005a)  algorithm  for  both 
the  training  (left)  and  testing  (right)  periods  of  the  driving  session  for  a  single  subject.  As 
expected,  strong  performance  was  observed  when  the  regression  model  is  reapplied  to  the 
training  set,  and  indeed,  this  was  observed  for  this  subject,  yielding  a  mean-squared  error  (MSE) 
measure  of  0.001  m  and  a  correlation  coefficient  of  0.89.  Performance  decreased  sharply  when 
the  regression  model  was  applied  to  the  testing  set  of  EEG  data,  producing  a  MSE  measure  of 
0.268  m  and  a  correlation  coefficient  of  0.31. 


Figure  9.  Actual  (black  line)  and  predicted  (blue  line)  lane  deviation  of  a  single  subject  using  the  global  form  of 
the  Lin  et  al.  (2005)  approach.  The  regression  algorithm  was  used  to  predict  lane  deviation  of  both  the 
training  data  (left)  and  the  testing  data  (right)  using  the  training  data. 

Table  1  relates  the  performance  of  the  algorithm  on  both  the  training  and  testing  data  for  all  the 
subjects.  In  general,  the  algorithm  performed  well  when  predicting  driver  performance  from 
EEG  signals  during  the  training  period  of  time.  For  the  14  subjects,  the  average  MSE  of  the 
prediction  was  0.002  ±  0.002  m,  with  an  average  correlation  between  predicted  and  actual  lane 
deviation  was  0.91  ±  0.06.  However,  when  the  regression  model  was  applied  to  portion  of  the 
driving  session  reserved  for  the  testing  period,  performance  declined  significantly  for  nearly 
every  driver.  Table  1  provides  the  prediction  accuracy  for  each  subject.  Across  the  population, 
the  average  MSE  of  the  prediction  for  the  population  was  0.335  ±  0.53  meters  with  an  average 
correlation  of  0.05  ±  0.25. 
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Table  1.  MSE  and  correlation  coefficients  of  lane  deviation  prediction  performance  for  the  global  and  local 
regression  algorithms. 


Driver 

Global  Regression 

Local  Regression 

Training 

Testing 

Training 

Testing 

MSE 

Corr  (R) 

MSE 

Corr  (R) 

MSE 

Corr  (R) 

MSE 

Corr  (R) 

1 

0.001 

0.909 

0.093 

-0.156 

0.002 

0.975 

0.008 

0.890 

2 

0.001 

0.890 

0.268 

0.313 

0.017 

0.930 

0.065 

0.743 

3 

0.001 

0.913 

0.073 

0.080 

0.001 

0.926 

0.003 

0.772 

4 

0.004 

0.921 

0.348 

-0.392 

0.029 

0.913 

0.110 

0.664 

5 

0.001 

0.931 

0.039 

-0.100 

0.001 

0.945 

0.007 

0.704 

6 

0.002 

0.905 

0.038 

0.295 

0.001 

0.926 

0.012 

0.586 

7 

0.001 

0.762 

1.740 

0.246 

0.001 

0.952 

0.009 

0.670 

8 

0.006 

0.791 

1.349 

0.540 

0.001 

0.924 

0.072 

0.371 

9 

0.003 

0.957 

0.076 

0.269 

0.002 

0.970 

0.013 

0.826 

10 

0.001 

0.963 

0.231 

-0.112 

0.001 

0.965 

0.025 

0.511 

11 

0.002 

0.967 

0.219 

-0.209 

0.010 

0.936 

0.130 

0.467 

12 

0.001 

0.965 

0.090 

-0.011 

0.004 

0.945 

0.145 

0.376 

13 

0.001 

0.942 

0.067 

0.019 

0.006 

0.911 

0.044 

0.503 

14 

0.0 

0.980 

0.057 

-0.053 

0.001 

0.957 

0.036 

0.407 

Average 

0.002 

0.914 

0.335 

0.052 

0.006 

0.941 

0.048 

0.606 

Clearly,  there  was  a  considerable  decrease  in  performance  in  this  approach  when  applied  to  the 
testing  data.  In  fact,  the  decrement  far  exceeded  that  observed  by  Lin  et  al.  (2005a)  for  their 
testing  data.  Thus,  it  is  likely  that  the  added  complexity  of  the  present  driving  simulation  is  at 
least  partly  responsible  for  poorer  regression  performance.  One  potential  explanation  for  this  is 
that  in  the  present  experiment,  changes  in  lane  deviation  may  be  related  to  subject- specific 
driving  habits  rather  than  a  lack  of  response  to  vehicle  perturbation  due  to  fatigue.  Another 
possible  explanation  is  that  the  nonstationary  nature  of  neural  signals  affects  or  changes  the 
connection  between  driver  performance  and  specific  elements  of  the  EEG  power  spectrum  over 
the  course  of  the  driving  session.  If  this  were  the  case,  one  would  expect  a  significant 
improvement  in  predictive  accuracy  when  the  local  regression  prediction  is  factored  in. 

Local  regression.  Figure  10  illustrates  the  measured  and  predicted  values  of  lane  deviation  of  the 
testing  data  using  the  local  form  of  the  Lin  et  al.  approach  for  the  same  subject  depicted  in  figure 
1.  A  comparison  of  the  figure  10A  with  the  right-hand  plot  of  figure  9  reveals  a  marked 
improvement  in  the  performance  of  the  local  vs.  the  global  for  the  testing  data  for  this  subject. 
This  difference  was  consistent  for  all  subjects,  with  the  local  algorithm  yielding  an  average  MSE 
error  of  0.006  ±  0.008  m  for  the  training  data  and  0.048  ±  0.048  m  for  the  testing  data,  and  with 
respective  correlations  between  predicted  and  actual  lane  deviation  of  0.94  ±  0.02  and 
0.61+0.17. 

Figure  10B  illustrates  the  relationship  between  the  MSE  of  the  prediction  in  global  vs.  local 
algorithms  for  all  14  subjects.  Figure  10C  demonstrates  a  similar  comparison  between 
approaches  for  the  correlations  of  the  predicted  and  the  actual  behavior.  While  the  global 
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regression  is  slightly  better  when  reapplied  to  the  training  data,  this  difference  was  not 
significant.  Conversely,  when  applied  to  the  testing  data,  both  the  MSE  error  values  and 
correlation  coefficients  were  significantly  different  between  the  two  algorithms  (p  <  0.05, 
Wilcoxon  test),  indicating  significantly  better  performance  of  the  local  regression  approach  for 
predicting  novel  data. 


Figure  10.  Comparison  of  local  and  global  regression  performance:  (A)  local 

regression  prediction  (red  line)  and  measured  values  (black  line)  of  lane 
deviation  across  the  driving  session  for  a  single  subject.  Also  visible  is  the 
trace  of  the  global  regression  prediction  (grey  line).  Global  vs.  local 
predictions  for  (B)  MSE  and  (C)  correlation  with  actual  driving  behavior 
for  all  subjects.  Open  circles  represent  the  training  data  and  closed  circles 
indicate  prediction  performance  on  testing  data. 

The  improved  performance  of  the  local  regression  algorithm  is  likely  due  to  the  relative 
instability  of  EEG  signals,  particularly  over  long  time  scales,  as  described.  Conversely,  because 
the  regression  coefficients  are  constantly  being  updated  to  match  more  recent  brain  activity  with 
driving  performance,  the  local  regression  algorithm  is  able  to  adapt  to  and  change  with  the 
transient  nature  of  EEG  signals  across  long  experimental  sessions. 

Note  that  the  local  approach  yields  a  noisier  prediction  of  driver  performance  (see  figure  10A), 
likely  due  to  an  enhanced  sensitivity  to  a  limited  number  of  EEG  features  that  may  have  only 
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transiently  or  artificially  improved  regression  during  the  shortened  training  period.  In  contrast, 
the  global  regression  algorithm  appears  less  sensitive  to  moment-to-moment  swings  in  EEG 
features,  resulting  in  a  somewhat  more  stable,  if  less  accurate  prediction  of  driver  performance. 

8.5  Conclusions 

The  accuracy  of  the  predicted  values  of  driver  performance  varied  greatly  between  drivers  for 
both  local  and  global  approaches.  Encouragingly,  regression  performance  on  training  data  was  in 
general  consistent  with  what  was  reported  in  a  similar  task,  however  somewhat  reduced, 
indicating  that  a  linear  regression  model  can  represent  the  data  reasonably  well.  On  the  other 
hand,  average  performance  on  testing  data  was  markedly  worse  in  this  study  than  was  reported 
by  the  algorithm's  original  developer.  However,  it  is  important  to  note  that  direct  comparison 
between  results  is  difficult,  as  we  did  not  enforce  the  same  exclusion  criteria  as  the  original 
investigators,  which  allowed  them  to  focus  only  on  those  subjects  who  exhibited  severe  fatigue. 
Instead,  we  considered  algorithm  performance  across  all  participating  subjects.  It  should  also  be 
mentioned  that  our  simulation  was  designed  to  be  more  reminiscent  of  actual  driving,  resulting  in 
a  more  variable  measure  of  driving  performance.  This  increased  variability  results  in  fluctuations 
in  the  performance  that  are  not  necessarily  attributable  to  fatigue  or  any  change  in  EEG  data  and 
potentially  adversely  affect  regression  performance. 

8.6  Future  Directions 

From  these  results,  it  appears  that  the  increased  complexity  of  naturalistic  driving  task  poses  a 
significant  challenge  to  EEG-based  driver  performance  prediction.  The  decrease  in  global 
regression  performance  for  the  testing  data  is  likely  due  to  the  increased  complexity  of 
continuously  controlling  vehicle  dynamics  in  this  experiment  vs.  that  of  Lin  et  al.  (2005a).  This 
is  a  troubling  trend,  however,  as  even  the  current  simulation  is  still  far  from  representative  of  the 
complex  environments  in  which  Soldiers  must  operate.  Nonetheless,  we  do  see  great  potential 
for  improving  upon  these  linear  regression  algorithms.  Toward  this  end,  accounting  for  the 
increased  variability  or  changes  in  performance  due  to  exogenous  factors  is  a  primary  concern. 
We  are  currently  investigating  several  avenues  we  have  identified  as  potential  means  to  advance 
the  concept  of  EEG-based  prediction  of  driver  performance: 

•  Adjusting  the  width  and/or  iteration  rate  of  the  moving  average  window  may  improve 
performance  by  optimizing  how  much  data  is  combined  into  a  given  data  point  used  by  the 
regression.  Increasing  the  width  and  iteration  rate  will  stabilize  EEG  estimates  over  time 
but  may  result  in  slower  estimation  of  changes  in  the  soldier/driver's  state. 

•  Combining  the  global  and  local  approaches  may  offer  improved  performance  over  either 
individually.  Each  approach,  global  and  local,  offer  distinct  advantages,  and  if  integrated 
appropriately  could  provide  a  more  accurate  and  stable  prediction  of  driver  performance. 
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One  approach  may  be  to  weight  them  based  on  their  covariances,  as  this  approach  may 
stem  the  influence  of  rapid  fluctuations  of  the  local  regression. 

•  Additional  or  alternative  behavioral  metrics  to  lane  deviation  are  being  considered.  For 
instance,  eye -tracking/blinking,  steering  control,  and  time-to-contact  of  lane  boundary  all 
have  a  solid  link  to  driver  fatigue,  but  might  also  be  more  strongly  tied  to  EEG  signals.  In 
addition,  these  measures  might  be  more  consistent  and  resilient  to  increases  in  task 
complexity  and  may  improve  predictive  performance. 

•  To  explore  improvements  to  the  regression  algorithm  itself,  SVR  is  also  being  used  to 
predict  driver  performance.  SVR  has  proven  to  be  a  highly  effective  and  easily 
implemented  machine-learning  approach,  and  is  likely  to  be  more  robust  to  variability  in 
the  driver  performance  metric  associated  with  more  complex  driving  conditions. 

•  Network  connectivity  measures  are  also  being  explored  to  assess  whether  driver 
performance  can  be  predicted  from  the  signals  from  a  combination  of  EEG  electrode  sites 
better  than  if  the  channels  considered  independently,  as  in  Lin’s  approach.  Further,  insights 
gained  from  this  analysis  may  also  be  useful  for  identifying  alternative  behavioral  metrics 
that  show  a  stronger  link  to  EEG  signals. 

•  Alternative  feature  extraction  methods  have  been  proposed  based  on  matching  pursuit  and 
wavelet  analysis.  Examining  specific  features  of  EEG  signals  extracted  using  these 
methods  may  improve  performance  by  feeding  only  the  most  relevant  signals  to  the 
regression  model. 

•  A  multitier  classification  of  driver  performance  may  provide  the  appropriate  level  of 
insight  into  driver  fatigue  while  also  improving  predictive  accuracy.  To  investigate  this 
potential,  SVC-based  approaches  to  predicting  driver  performance  from  EEG  signals  are 
also  being  explored.  An  additional  advantage  of  this  tact  is  that  by  condensing  performance 
into  classes,  we  reduce  a  great  deal  of  variability  in  the  performance  metric  that 
accompanies  increases  in  task  complexity. 

In  addition  to  the  aforementioned  approaches,  we  are  also  continuing  to  work  with  the 
developers  of  the  original  algorithm.  They  too  have  explored  alternative  approaches,  focusing 
largely  on  algorithms  involving  Independent  Component  Analysis  (ICA)  (Lin  et  al.,  2005b; 
Chuang  et  al.,  2012).  ICA  offers  a  means  to  focus  on  a  specific,  potentially  more  salient  source 
of  neural  activity  than  channel  data. 

Given  the  encouraging  results  of  the  original  Lin  et  al.  (2005a)  approach  as  applied  here,  we 
believe  that  these  proposed  extensions  are  likely  to  facilitate  the  expansion  of  their  basic  method 
to  increasingly  more  complex  driving  environments.  It  is  our  further  hope  that  the  resulting 
system  will  have  application  beyond  the  vehicle,  capable  of  reliably  predicting  lapses  in 
performance  in  any  task  which  requires  an  alert  and  attentive  Soldier  or  agent. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 

CaN-CTA  Cognition  and  Neuroergonomics  Collaborative  Technology  Alliance 

CATS  cognitive  avionics  tool  set 

CTT  continuous  tracking  task 

DARPA  Defense  Advanced  Research  Projects  Agency 

EDA  electrodermal  activity 

EEG  electroencephalography 

EKG  electrocardiography 

EMI  electromagnetic  interference 

fMRI  functional  magnetic  resonance  imaging 

FMTs  fatigue  management  technologies 

GVSL  ground  vehicle  simulation  laboratory 

ICA  independent  component  analysis 

ILIR  in-house  laboratory  independent  research 

KSS  Karolinska  Sleepiness  Scale 

MEG  magnetoencephalography 

NCCF  nickel-coated  carbon  fibers 

NCTU  National  Chiao  Tung  University 

NIH  National  Institute  for  Health 

PCA  principal  component  analysis 

PDMS  poly-dimethyl  siloxane 

POC  Physical  Optics  Corp 

PVT  psychomotor  vigilance  task 

RMS  ride  motion  simulator 

SBIR  small  business  innovative  research 

TSD  temporal  synchronization  device 
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(PDF)  RDRL  HRM  CU  B  LUT AS -SPENCER 
6501  E  11  MILE  RD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  AV  W  CULBERTSON 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  HUMAN  RSRCH  AND  ENGRNG 

DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  C  CARSTENS 
6450  WAY  ST 
BLDG  2839  RM  310 
FORT  BENNING  GA  31905-5400 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  RDRL  HRM  DE  A  MARES 

1733  PLEASONTON  RD  BOX  3 
FORT  BLISS  TX  79916-6816 
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COPIES  ORGANIZATION 


8  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  SIMULATION  &  TRAINING 
TECHNOLOGY  CENTER 
RDRL  HRT  COL  M  CLARKE 
RDRL  HRT  I  MARTINEZ 
RDRL  HRT  T  R  SOTTILARE 
RDRL  HRT  B  N  FINKELSTEIN 
RDRL  HRT  G  A  RODRIGUEZ 
RDRL  HRT  I  I  HART 
RDRL  HRT  M  C  METEVIER 
RDRL  HRT  S  B  PETTIT 
12423  RESEARCH  PARKWAY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
(PDF)  (PDF)  HQ  USASOC 

RDRL  HRM  CN  R  SPENCER 
BLDG  E2929  DESERT  STORM  DRIVE 
FORT  BRAGG  NC  28310 

1  ARMY  G1 
(PDF)  DAPE  MR  B  KNAPP 

300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 

ABERDEEN  PROVING  GROUND 

13  DIRUSARL 
(PDF)  RDRL  HR 

L  ALLENDER 
P  FRANASZCZUK 
C  COSENZO 
RDRL  HRM 

P  S  A V AGE-KNEPS  HIELD 
RDRL  HRM  AL 
C  PAULILLO 
RDRL  HRM  B 
C  SAMMS 
RDRL  HRM  C 
L  GARRETT 
RDRL  HRS 
J  LOCKETT 
RDRL  HRS  B 
M  LAFIANDRA 
RDRL  HRS  C 
B  LANCE 
K  MCDOWELL 
RDRL  HRS  D 
B  AMREIN 
RDRL  HRS  E 
D  HEADLEY 
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